You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "chewbranca (via GitHub)" <gi...@apache.org> on 2025/07/18 18:48:42 UTC

[PR] Couch stats resource tracker v3 rebase main [couchdb]

chewbranca opened a new pull request, #5602:
URL: https://github.com/apache/couchdb/pull/5602

   This PR supercedes https://github.com/apache/couchdb/pull/5491 and includes @iilyak's excellent HTTP updates on top of it, as well as some final cleanup and documentation from me. I've copied the contents of `CSRT.md` into the PR description here:
   
   # Couch Stats Resource Tracker (CSRT)
   
   CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
   tracks the quantity of resources induced at the process level in a live
   queryable manner that also generates process lifetime reports containing
   statistics on the total resource load of a request, as a function of things like
   dbs/docs opened, view and changes rows read, changes returned vs processed,
   Javascript filter usage, duration, and more. This system is a paradigm shift in
   CouchDB visibility and introspection, allowing for expressive real time querying
   capabilities to introspect, understand, and aggregate CouchDB internal resource
   usage, as well as powerful filtering facilities for conditionally generating
   reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
   `recon:proc_window` with `csrt:proc_window` allowing for the same style of
   battle hardened introspection with Recon's excellent `proc_window`, but with the
   sample window over any of the CSRT tracked CouchDB stats!
   
   CSRT does this by piggy-backing off of the existing metrics tracked by way of
   `couch_stats:increment_counter` at the time when the local process induces those
   metrics inc calls, and then CSRT updates an ets entry containing the context
   information for the local process, such that global aggregate queries can be
   performed against the ets table as well as the generation of the process
   resource usage reports at the conclusions of the process's lifecyle.The ability
   to do aggregate querying in realtime in addition to the process lifecycle
   reports for post facto analysis over time, is a cornerstone of CSRT that is the
   result of a series of iterations until a robust and scalable aproach was built.
   
   The real time querying is achieved by way of a global ets table with
   `read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
   Great care was taken to ensure that _zero_ concurrent writes to the same key
   occure in this model, and this entire system is predicated on the fact that
   incremental updates to `ets:update_counters` provides *really* fast and
   efficient updates in an atomic and isolated fashion when coupled with
   decentralized counters and write concurrency. Each process that calls
   `couch_stats:increment_counter` tracks their local context in CSRT as well, with
   zero concurrent writes from any other processes. Outside of the context setup
   and teardown logic, _only_ operations to `ets:update_counter` are performed, one
   per process invocation of `couch_stats:increment_counter`, and one for
   coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
   of ets calls to real time stats updates for the primary workloads.
   
   The primary achievement of CSRT is the core framework iself for concurrent
   process local stats tracking and real time RPC delta accumulation in a scalable
   manner that allows for real time aggregate querying and process lifecycle
   reports. This took several versions to find a scalable and robust approach that
   induced minimal impact on maximum system throughput. Now that the framework is
   in place, it can be extended to track any further desired process local uses of
   `couch_stats:increment_counter`. That said, the currently selected set of stats
   to track was heavily influenced by the challenges in reotractively understanding
   the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
   similarly, `/db/_find`.
   
   CSRT started as an extension of the Mango execution stats logic to `_changes`
   feeds to get proper visibility into quantity of docs read and filtered per
   changes request, but then the focus inverted with the realization that we should
   instead use the existing stats tracking mechanisms that have already been deemed
   critical information to track, which then also allows for the real time tracking
   and aggregate query capabilities. The Mango execution stats can be ported into
   CSRT itself and just become one subset of the stats tracked as a whole, and
   similarly, any additional desired stats tracking can be easily added and will
   be picked up in the RPC deltas and process lifetime reports.
   
   # CSRT Config Keys
   
   ## -define(CSRT, "csrt").
   
   > config:get("csrt").
   
   Primary CSRT config namespace: contains core settings for enabling different
   layers of functionality in CSRT, along with global config settings for limiting
   data volume generation.
   
   ## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
   
   > config:get("csrt_logger.matchers_enabled").
   
   Config toggles for enabling specific builtin logger matchers, see the dedicated
   section below on `# CSRT Default Matchers`.
   
   ## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
   
   > config:get("csrt_logger.matchers_threshold").
   
   Config settings for defining the primary `Threshold` value of the builtin logger
   matchers, see the dedicated section below on `# CSRT Default Matchers`.
   
   ## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
   
   > config:get("csrt_logger.matchers_enabled").
   
   Config section for setting `$db_name = $threshold` resulting in instantiating a
   "dbname_io" logger matcher for each `$db_name` that will generate a CSRT
   lifecycle report for any contexts that that induced more operations on _any_ one
   field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
   than `$threshold` and is on database `$db_name`.
   
   This is basically a simple matcher for finding heavy IO requests on a particular
   database, in a manner amenable to key/value pair specifications in this .ini
   file until a more sophisticated declarative model exists. In particular, it's
   not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
   alternative mechanism for either dynamically assembling an `#rctx{}` to match
   against or generating the raw matchspecs themselves is warranted.
   
   ## -define(CSRT_INIT_P, "csrt.init_p").
   
   > config:get("csrt.init_p").
   
   Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
   way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
   for the desired RPC operations in an expandable manner, without having to add
   new stats for every single potential RPC operation. These are for the individual
   metrics to track, the feature is enabled by way of the config toggle
   `config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
   most part until new operations are tracked.
   
   # CSRT Code Markers
   
   ## -define(CSRT_ETS, csrt_server).
   
   This is the reference to the CSRT ets table, it's managed by `csrt_server` so
   that's where the name originates from.
   
   ## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
   
   This marker is where the active matchers are written to in `persisten_term` for
   concurrently and parallelly and accessing the logger matchers in the CSRT
   tracker processes for lifecycle reporting.
   
   # CSRT Process Dictionary Markers
   
   ## -define(PID_REF, {csrt, pid_ref}).
   
   This marker is for the core storing the core `PidRef` identifier. The key idea
   here is that a lifecycle is a context lifecycle is contained to within the given
   `PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
   pass those to different workers.
   
   This is specifically necessary for long running processes that need to handle
   many CSRT context lifecycles over the course of that individual process's
   lifecycle independent. In practice, this is immediately needed for the actual
   coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
   handlers that can be re-used, so we need a way to create a CSRT lifecycle
   corresponding to the given request currently being serviced. This is also
   intended to be used in other long running processes, like IOQ or `couch_js` pids
   such that we can track the specific context inducing the operations on the
   `couch_file` pid or indexer or replicator or whatever.
   
   Worker processes have a more clear cut lifecycle, but either style of process
   can be exit'ed in a manner that skips the ability to do cleanup operations, so
   additionally there's a dedicated tracker process spawned to monitor the process
   that induced the CSRT context such that we can do the dynamic logger matching
   directly in these tracker processes and also we can properly cleanup the ets
   entries even if the Pid crashes.
   
   ## -define(TRACKER_PID, {csrt, tracker}).
   
   A handle to the spawned tracker process that does cleanup and logger matching
   reprots at the end of the process lifecycle. We store a reference to the tracker
   pid so that for explicit context destruction, like in `chttpd` workers after a
   request has been serviced, we can update stop the tracker and perform the
   expected cleanup directly.
   
   ## -define(DELTA_TA, {csrt, delta_ta}).
   
   This stores our last delta snapshot to track progress since the last incremental
   streaming of stats back to the coordinator process. This will be updated after
   the next delta is made with the latest value. Eg this stores `T0` so we can do
   `T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
   for use in our next delta.
   
   ## -define(LAST_UPDATED, {csrt, last_updated}).
   
   This stores the integer corresponding to the `erlang:monotonic_time()` value of
   the most recent `updated_at` value. Basically this lets us utilize a pdict
   value to be able to turn `update_at` tracking into an incremental operation that
   can be chained in the existing atomic `ets:update_counter` and
   `ets:update_element` calls.
   
   The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
   which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
   strict use of the atomic operations for tracking these values is why this
   system works effeciently at scale. This means that we can increment counters on
   all of the stats counter fields in a batch, very quickly, but for tracking
   `updated_at` timestamps we'd need to either do an extra ets call to get the last
   `updated_at` value, or do an extra ets call to `ets:update_element` to set the
   `updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
   batch inc operation is essentially the only write operation performed after the
   initial context setting of dbname/handler/etc; this means that we'd literally
   double the number of ets calls induced to track CSRT updates, just for tracking
   the `updated_at`. So instead, we rely on the fact that the local process
   corresponding to `$pid_ref` is the _only_ process doing updates so we know the
   last `updated_at` value will be the last time this process updated the data. So
   we track that value in the pdict and then take a delta between `tnow()` and
   `updated_at`, and then `updated_at` becomes a value we can sneak into the other
   integer counter updates we're already performing!
   
   # Primary Config Toggles
   
   # CSRT (?CSRT="csrt") Config Settings
   
   ## config:get(?CSRT, "enable", false).
   
   Core enablement toggle for CSRT, defaults to false. Enabling this setting
   intiates local CSRT stats collection as well as shipping deltas in RPC
   responses to accumulate in the coordinator.
   
   This does _not_ trigger the new RPC spawn metrics, and it does not enable
   reporting for any of the rctx types.
   
   *NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
   _before_ you enable it on any node, otherwise the old version nodes won't know
   how to handle the new RPC formats including an embedded Delta payload.
   
   ## config:get(?CSRT, "enable_init_p", false).
   
   Enablement of tracking new metric counters for different `fabric_rpc` operations
   types to track spawn rates of RPC work induced across the cluster. There is
   corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
   form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
   for enabling the specific RPC endpoints.
   
   However, those individual settings can be ignored and this top level config
   toggle is what should be used in general, as the function specific config
   toggles predominantly exist to enable tracking a subet of total RPC operations
   in the cluster, and new endpoints can be added here.
   
   ## config:get(?CSRT, "enable_reporting", false).
   
   This is the primary toggle for enabling CSRT process lifetime reports containing
   detailed information about the quantity of work induced by the given
   request/worker/etc. This is the top level toggle for enabling _any_ reporting,
   and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
   disable the reporting of any individual RPC workers, leaving the coordinator
   responsible of generating a report with the accumulated deltas.
   
   ## config:get(?CSRT, "enable_rpc_reporting", false).
   
   This enables the possibility of RPC workers generating reports. They still need
   to hit the configured thresholds to induce a report, but this will generate CSRT
   process lifetime reports for individual RPC workers that trigger the configured
   logger thresholds. This allows for quantifying per node resource usage when
   desired, as otherwise the reports are at the http request level and don't
   provide per node stats.
   
   The key idea here is that having RPC level CSRT process lifetime reporting is
   incredibly useful, but can also generate large quantities of data. For example,
   a view query on a Q=64 database will stream results from 64 shard replicas,
   resulting in at least 64 RPC reports, plus any that might have been generated
   from RPC workers that "lost" the race for shard replica. This is very useful,
   but a lot of data given the verbose nature of funneling it through the RSyslog
   reports, however, the ability to write directly to something like ClickHouse or
   another columnar store would be great.
   
   Until there's an efficient storage mechanism to stream the results to, the
   rsyslog entries work great and are very practical, but care must be taken to
   not generate too much data for aggregate queries as they generate at least `Qx`
   more report than an individual report per http request from the coordinator.
   This setting exists as a way to either a) utilize the logger matcher configured
   thresholds to allow for _any_ rctx's to be recorded when they induce heavy
   operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
   the coordinator level.
   
   NOTE: this setting exists because we lack an expressive enough config
   declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
   special compile time parse transform macro that requires the fully definition to
   be specified directly, it cannot be iteractively constructed. That said, you
   _can_ register matchers through remsh with more specific and fine grained
   pattern matching, and a more expressive system for defining matchers are being
   explored.
   
   ## config:get_boolean(?CSRT, "should_truncate_reports", true)
   
   Enables truncation of the CSRT process lifetime reports to not include any
   fields that are zero at the end of process lifetime, eg don't include
   `js_filter=0` in the report if the request did not induce Javascript filtering.
   
   This can be disabled if you really care about consistent fields in the report
   logs, but this is a log space saving mechanism, similar to disabling RPC
   reporting by default, as its a simple way to reduce overall volume
   
   ## config:get(?CSRT, "randomize_testing", true).
   
   This is a `make eunit` only feature toggle that will induce randomness into the
   cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
   exercise edge case scenarios and failures when CSRT is only conditionally
   enabled, ensuring that it gracefuly and robustly handles errors without fallout
   to the underlying http clients.
   
   The idea here is to introduce randomness into whether CSRT is enabled across all
   the nodes to simulate clusters with heterogeneous CSRT enablement and also to
   ensure that CSRT works properly when toggled on/off wihout causing any
   unexpected fallout to the client requests.
   
   This is a config toggle specifically so that the actual CSRT tests can disable
   it for making accurate assertions about resource usage traacking, and is not
   intended to be used directly.
   
   ## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
   
   Limit the quantity of rows that can be loaded in an http query.
   
   # CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
   
   ## config:get(?CSRT_INIT_P, ModFunName, false).
   
   These config toggles exist to conditionaly enable additional tracking of RPC
   endpoints of interest, but rather it's a way to selectively enable tracking for
   a subset of RPC operations, in a way we can extend later to add more. The
   `ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
   `"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
   
   NOTE: this is a bit awkward and isn't meant to be used directly, instead,
   utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
   as a whole.
   
   The current set of operations, as copied in from `default.ini`
   
   ```
   [csrt.init_p]
   fabric_rpc__all_docs = true
   fabric_rpc__changes = true
   fabric_rpc__get_all_security = true
   fabric_rpc__map_view = true
   fabric_rpc__open_doc = true
   fabric_rpc__open_shard = true
   fabric_rpc__reduce_view = true
   fabric_rpc__update_docs = true
   ```
   
   # CSRT Logger Matcher Enablement and Thresholds
   
   There are currently six builtin default loggers designed to make it easy to do
   filtering on heavy resource usage inducing and long running requests. These are
   designed as a simple baseline of useful matchers, declared in a manner amenable
   to `default.ini` based constructs. More expressive matcher declarations are
   being explored, and matchers of arbitrary complexity can be registered directly
   through remsh. The default matchers are all designed around an integer config
   Threshold that triggers on a specific field, eg docs read, or on a delta of
   fields for long requests and changes requests that process many rows but return
   few.
   
   The current default matchers are:
   
     * docs_read: match all requests reading more than N docs
     * rows_read: match all requests reading more than N rows
     * docs_written: match all requests writing more than N docs
     * long_reqs: match all requests lasting more than N milliseconds
     * changes_processed: match all changes requests that returned at least N rows
       less than was necessarily loaded to complete the request (eg find heavy
       filtered changes requests reading many rows but returning few).
     * ioq_calls: match all requests inducing more than N ioq_calls
   
   Each of the default matchers has an enablement setting in
   `config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
   corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
   Name)` that is an integer value corresponding to the specific nature of that
   matcher.
   
   ## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
   
   > -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
   
   ### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
   
   Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
   that any request that reads more than `Threshold` docs will generate a CSRT
   process lifetime report with a summary of its resouce consumption.
   
   This is different from the `rows_read` filter in that a view with `?limit=1000`
   will read 1000 rows, but the same request with `?include_docs=true` will also
   induce an additional 1000 docs read.
   
   ### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
   
   Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
   that any request that reads more than `Threshold` rows will generate a CSRT
   process lifetime report with a summary of its resouce consumption.
   
   This is different from the `docs_read` filter so that we can distinguish between
   heavy view requests with lots of rows or heavy requests with lots of docs.
   
   ### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
   
   Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
   that any request that writtens more than `Threshold` docs will generate a CSRT
   process lifetime report with a summary of its resouce consumption.
   
   ### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
   
   Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
   that any request that induces more than `Threshold` IOQ calls will generate a
   CSRT process lifetime report with a summary of its resouce consumption.
   
   ### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
   
   Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
   that any request where the the last CSRT rctx `updated_at` timestamp is at least
   `Threshold` milliseconds grather than the `started_at timestamp` will generate a
   CSRT process lifetime report with a summary of its resource consumption.
   
   ## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
   
   > -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
   
   ### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
   
   Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
   
   ### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
   
   Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
   
   ### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
   
   Threshold for `docs_written` logger matcher, defaults to `500` docs written.
   
   ### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
   
   Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
   
   ### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
   
   Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
   
   # Core CSRT API
   
   The `csrt(.erl)` module is the primary entry point into CSRT, containing API
   functionality for tracking the lifecycle of processes, inducing metric tracking
   over that lifecycle, and also a variety of functions for aggregate querying.
   
   It's worth noting that the CSRT context tracking functions are specifically
   designed to not `throw` and be safe in the event of unexpected CSRT failures or
   edge cases. The aggregate query API has some callers that will actually throw,
   but aside from this core CSRT operations will not bubble up exceptions, and will
   either return the error value, or catch the error and move on rather than
   chaining further errors.
   
   ## PidRef API
   
   These are functions are CRUD operations around creating and storing the CSRT
   `PidRef` handle.
   
   ```
   -export([
       destroy_pid_ref/0,
       destroy_pid_ref/1,
       create_pid_ref/0,
       get_pid_ref/0,
       get_pid_ref/1,
       set_pid_ref/1
   ]).
   ```
   
   ## Context Lifecycle API
   
   These are the CRUD functions for handling a CSRT context lifecycle, where a
   lifecycle context is created in a `chttpd` coordinator process by way of
   `csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
   `csrt:create_worker_context/3`. Additional functions are exposed for setting
   context specific info like username/dbname/handler. `get_resource` fetches the
   context being tracked corresponding to the given `PidRef`.
   
   ```
   -export([
       create_context/2,
       create_coordinator_context/2,
       create_worker_context/3,
       destroy_context/0,
       destroy_context/1,
       get_resource/0,
       get_resource/1,
       set_context_dbname/1,
       set_context_dbname/2,
       set_context_handler_fun/1,
       set_context_handler_fun/2,
       set_context_username/1,
       set_context_username/2
   ]).
   ```
   
   ## Public API
   
   The "Public" or miscellaneous API for lack of a better name. These are various
   functions exposed for wider use and/or testing purposes.
   
   ```
   -export([
       clear_pdict_markers/0,
       do_report/2,
       is_enabled/0,
       is_enabled_init_p/0,
       maybe_report/2,
       to_json/1
   ]).
   ```
   
   ## Stats Collection API
   
   This is the stats collection API utilized by way of
   `couch_stats:increment_counter` to do local process tracking, and also in `rexi`
   to adding and extracting delta contexts and then accumulating those values.
   
   NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
   by way of the last local pdict's rctx delta snapshot, and then update to the
   most recent version. Two individual rctx snapshots for a PidRef can safely
   generate an actual delta by way of `csrt_util:rctx_delta/2`.
   
   ```
   -export([
       accumulate_delta/1,
       add_delta/2,
       docs_written/1,
       extract_delta/1,
       get_delta/0,
       inc/1,
       inc/2,
       ioq_called/0,
       js_filtered/1,
       make_delta/0,
       rctx_delta/2,
       maybe_add_delta/1,
       maybe_add_delta/2,
       maybe_inc/2,
       should_track_init_p/1
   ]).
   ```
   
   ## TODO: RPC/QUERY DOCS
   
   ```
   %% RPC API
   -export([
       rpc/2,
       call/1
   ]).
   
   %% Aggregate Query API
   -export([
       active/0,
       active/1,
       active_coordinators/0,
       active_coordinators/1,
       active_workers/0,
       active_workers/1,
       count_by/1,
       find_by_nonce/1,
       find_by_pid/1,
       find_by_pidref/1,
       find_workers_by_pidref/1,
       group_by/2,
       group_by/3,
       query/1,
       query/2,
       query_matcher/1,
       query_matcher/2,
       sorted/1,
       sorted_by/1,
       sorted_by/2,
       sorted_by/3
   ]).
   ```
   
   ## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
   
   This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
   `proc_window` style aggregations/sorting/filtering but with the stats fields
   collected by CSRT! This is also a direct port of `recon:proc_window` in that it
   utilizes the same underlying logic and effecient internal data structures as
   `recon:proc_window`, but rather only changes the Sample function:
   
   ```erlang
   %% This is a recon:proc_window/3 [1] port with the same core logic but
   %% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
   %% pid_ref() rather than pid().
   %% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
   -spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
       AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
   proc_window(AttrName, Num, Time) ->
       Sample = fun() -> pid_ref_attrs(AttrName) end,
       {First, Last} = recon_lib:sample(Time, Sample),
       recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
   ```
   
   In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
   and in fact, if recon upstream parameterized the option of `AttrName` or
   `SampleFunction`, this could be reimplemented as:
   
   ```erlang
   %% csrt:proc_window
   proc_window(AttrName, Num, Time) ->
       Sample = fun() -> pid_ref_attrs(AttrName) end,
       recon:proc_window(Sample, Num, Time).
   ```
   
   This implementation is being highlighted here because `recon:proc_window/3` is
   battle hardened and `recon_lib:sliding_window` uses an effecient internal data
   structure for storing the two samples that has been proven to work in production
   systems with millions of active processes, so swapping the `Sample` function
   with a CSRT version allows us to utilize the production grade recon
   functionality, but extended out to the particular CouchDB statistics we're
   esepecially interested in.
   
   And on a fun note: any further stats tracking fields added to CSRT tracking will
   automatically work with this too.
   
   
   ```
   -export([
       pid_ref_attrs/1,
       pid_ref_matchspec/1,
       proc_window/3
   ]).
   ```
   
   <hr />
   
   # Core types and Maybe types
   
   Before we look at the `#rctx{}` record fields, lets examine the core datatypes
   defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
   essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
   
   Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
   undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
   maybe_foo()` and then we can use Dialyzer to statically assert all callers of
   `get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
   ensure that all subsequent callers do as well.
   
   This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
   throughout CSRT and has greatly aided in the development, refactoring, and
   static analysis of this system. Here's a useful snippet for running Dialyzer
   while hacking on CSRT:
   
   > make && time make dialyze apps=couch_stats
   
   ```erlang
   -type pid_ref() :: {pid(), reference()}.
   -type maybe_pid_ref() :: pid_ref() | undefined.
   
   -type coordinator_rctx() :: #rctx{type :: coordinator()}.
   -type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
   -type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
   -type rctxs() :: [#rctx{}] | [].
   -type maybe_rctx() :: rctx() | undefined.
   ```
   
   Above we have the core `pid_ref()` data type, which is just a tuple with a
   `pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
   optional presence of a `pid_ref()`, allowing for our APIs like
   `csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
   `pid_ref()`.
   
   We define our core `rctx()` data type as an empty `#rctx{}`, or the more
   specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
   specific about the `rctx()` type in functions that need to distinguish. And then
   as expected, we have the notion of `maybe_rctx()`.
   
   # #rctx{}
   
   This is the core data structure utilized to track a CSRT context for a
   coordinator or rpc_worker process, represented by the `#rctx{}` record, and
   stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
   
   The Metadata fields store labeling data for the given process being tracked,
   such as started_at and updated_at timings, the primary `pid_ref` id key, the
   type of the process context, and some additional information like username,
   dbname, and the nonce of the coordinator request.
   
   The Stats Counters fields are `non_neg_integer()` monotonically increasing
   counters corresponding to the `couch_stats` metrics counters we're interested in
   tracking at a process level cardinality. The use of these purely integer counter
   fields represented by a record represented in an ets table is the cornerstone of
   CSRT and why its able to operate at high throughput and high concurrency, as
   `ets:update_counter/{3,4}` take increment operations to be performed atomically
   and in isolation, in a manner in which does not require fetching and loading the
   data directly. We then take care to batch the accumulation of delta updates into
   a single `update_counter` call and even sneak in the `updated_at` tracking as a
   integer counter update without inducing an extra ets call.
   
   NOTE: the typespec's for these fields include `'_'` atoms as possible types as
   that is the matchspec wildcard any of the fields can be set to when using an
   existing `#rctx{}` record to search with.
   
   
   ```erlang
   -record(rctx, {
       %% Metadata
       started_at = csrt_util:tnow() :: integer() | '_',
       %% NOTE: updated_at must be after started_at to preserve time congruity
       updated_at = csrt_util:tnow() :: integer() | '_',
       pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
       nonce :: nonce() | undefined | '_',
       type :: rctx_type() | undefined | '_',
       dbname :: dbname() | undefined | '_',
       username :: username() | undefined | '_',
   
       %% Stats Counters
       db_open = 0 :: non_neg_integer() | '_',
       docs_read = 0 :: non_neg_integer() | '_',
       docs_written = 0 :: non_neg_integer() | '_',
       rows_read = 0 :: non_neg_integer() | '_',
       changes_returned = 0 :: non_neg_integer() | '_',
       ioq_calls = 0 :: non_neg_integer() | '_',
       js_filter = 0 :: non_neg_integer() | '_',
       js_filtered_docs = 0 :: non_neg_integer() | '_',
       get_kv_node = 0 :: non_neg_integer() | '_',
       get_kp_node = 0 :: non_neg_integer() | '_'
       %% "Example to extend CSRT"
       %%write_kv_node = 0 :: non_neg_integer() | '_',
       %%write_kp_node = 0 :: non_neg_integer() | '_'
   }).
   ```
   
   ## Metadata
   
   We use `csrt_util:tnow()` for time tracking, which is a `native` format
   `erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
   negative value. You must either take a delta or convert the time to get into a
   useable format, as one might suspect by the use of `native`.
   
   We make use of `erlang:mononotic_time/0` as per the recommendation in
   https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
   for the suggested way to `Measure Elasped Time`, as quoted:
   
   ```
   Take time stamps with erlang:monotonic_time/0 and calculate the time difference
   using ordinary subtraction. The result is in native time unit. If you want to
   convert the result to another time unit, you can use erlang:convert_time_unit/3.
   
   An easier way to do this is to use erlang:monotonic_time/1 with the desired time
   unit. However, you can then lose accuracy and precision.
   ```
   
   So our `csrt_util:tnow/0` is implemented as the following, and we store
   timestamps in `native` format as long as possible to avoid precision loss at
   higher units of time, eg 300 microseconds is zero milliseconds.
   
   ```
   -spec tnow() -> integer().
   tnow() ->
       erlang:monotonic_time().
   ```
   
   We store timestamps in the node's local erlang representation of time,
   specifically to be able to effeciently do time deltas, and then we track time
   deltas from the local node's perspective to not send timestamps across the wire.
   We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
   native time representation to its corresponding time format when we generate the
   process life cycle reports or send an http response.
   
   NOTE: because we do an inline definition and assignment of the
   `#rctx.started_at` and `#rctx.updated_at` fields to `csrt_util:tnow()`, we
   _must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
   fundamental time incongruenties.
   
   ### #rctx.started_at = csrt_util:tnow() :: integer() | '_',
   
   A static value corresponding to the local node's Erlang monotonic_time at which
   this context was created.
   
   ### #rctx.updated_at = csrt_util:tnow() :: integer() | '_',
   
   A dynamic value corresponding to the local node's Erlang monotonic_time at which
   this context was updated. Note: unlike `#rctx.started_at`, this value will
   update over time, and in the process lifecycle reports the `#rctx.updated_at`
   value corresponds to the point at which the context was destroyed, allowing for
   calculation of the total duration of the request/context.
   
   ### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
   
   The primary identifier used to track the resources consumed by a given `pid()`
   for a specific context identified with a `make_ref()`, and combined together as
   unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
   over time.
   
   ### #rctx.nonce :: nonce() | undefined | '_',
   
   The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
   used as the primary grouping identifier of workers across the cluster, as the
   `Nonce` is funneled through `rexi_server`.
   
   ### #rctx.type :: rctx_type() | undefined | '_',
   
   A subtype classifier for the `#rctx{}` contexts, right now only supporting
   `#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
   additional context types like `#view_indexer{}`, `#search_indexer{}`,
   `#replicator{}`, `#compactor{}`, `#etc{}`.
   
   ### #rctx.dbname :: dbname() | undefined | '_',
   
   The database name, filled in at some point after the initial context creation by
   way of `csrt:set_context_dbname/{1,2}`.
   
   ### #rctx.username :: username() | undefined | '_',
   
   The requester's username, filled in at some point after the initial context
   creation by way of `csrt:set_context_username/{1,2}`.
   
   ## Stats Counters
   
   All of these stats counters are stricly `non_neg_integer()` counter values that
   are monotonically increasing, as we only induce positive counter increment calls
   in CSRT. Not all of these values will be nonzero, eg if the context doesn't
   induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
   field. The `"should_truncate_reports"` config value described in this document
   will conditionally exclude the zero valued fields from being included in the
   process life cycle report.
   
   ### #rctx.db_open = 0 :: non_neg_integer() | '_',
   
   > Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
   
   The number of `couch_server:open/2` invocations induced by this context.
   
   ### #rctx.docs_read = 0 :: non_neg_integer() | '_',
   
   > Tracking `couch_stats:increment_counter([couchdb, database_reads])
   
   The number of `couch_db:open_doc/3` invocations induced by this context.
   
   ### #rctx.docs_written = 0 :: non_neg_integer() | '_',
   
   A phony metric counting docs written by the context, induced by
   `csrt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
   count the magnitude of docs written, as the actual document writes happen in the
   `#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
   supported in CSRT.
   
   This can be replaced with direct counting  once passthrough contexts work.
   
   ### #rctx.rows_read = 0 :: non_neg_integer() | '_',
   
   > Tracking `couch_stats:increment_counter([fabric_rpc, changes, processed])
   > also Tracking `couch_stats:increment_counter([fabric_rpc, view, rows_read])
   
   A value tracking multiple possible metrics corresponding to rows streamed in
   aggregate operations. This is used for view_rows/changes_rows/all_docs/etc.
   
   ### #rctx.changes_returned = 0 :: non_neg_integer() | '_',
   
   The number of `fabric_rpc:changes_row/2` invocations induced by this context,
   specifically tracking the number of changes rows streamed back to the client
   requeest, allowing for distinguishing between the number of changes processed to
   fulfill a request versus the number actually returned in the http response.
   
   ### #rctx.ioq_calls = 0 :: non_neg_integer() | '_',
   
   A phony metric counting invocations of `ioq:call/3` induced by this context. As
   with `#rctx.docs_written`, we need a proxy metric to reperesent these calls
   until CSRT context passing is supported so that the `ioq_server` pid and return
   its own delta back to the worker pid.
   
   ### #rctx.js_filter = 0 :: non_neg_integer() | '_',
   
   A phony metric counting the number of `couch_query_servers:filter_docs_int/5`
   (eg ddoc_prompt) invocations induced by this context. This is called by way of
   `csrt:js_filtered(length(JsonDocs))` which both increments `js_filter` by 1, and
   `js_filtered_docs` by the length of the docs so we can track magnitude of docs
   and doc revs being filtered.
   
   ### #rctx.js_filtered_docs = 0 :: non_neg_integer() | '_',
   
   A phony metric counting the quantity of documents filtered by way of
   `couch_query_servers:filter_docs_int/5` (eg ddoc_prompt) invocations induced by
   this context. This is called by way of `csrt:js_filtered(length(JsonDocs))`
   which both increments `#rctx.js_filter` by 1, and `#rctx.js_filtered_docs` by
   the length of the docs so we can track magnitude of docs and doc revs being
   filtered.
   
   ### #rctx.get_kv_node = 0 :: non_neg_integer() | '_',
   
   This metric tracks the number of invocations to `couch_btree:get_node/2` in
   which the `NodeType` returned by `couch_file:pread_term/2` is `kv_node`, instead
   of `kp_node`.
   
   This provides a mechanism to quantify the impact of document count and document
   size as those values become larger in the logarithmic complexity btree
   algorithms.  size on the logarithmic complexity btree algorithms as the database
   btrees grow.
   
   ### #rctx.get_kp_node = 0 :: non_neg_integer() | '_'
   
   This metric tracks the number of invocations to `couch_btree:get_node/2` in
   which the `NodeType` returned by `couch_file:pread_term/2` is `kp_node`, instead
   of `kv_node`.
   
   This provides a mechanism to quantify the impact of document count and document
   size as those values become larger in the logarithmic complexity btree
   algorithms.  size on the logarithmic complexity btree algorithms as the database
   btrees grow.
   
   %% "Example to extend CSRT"
   %%write_kv_node = 0 :: non_neg_integer() | '_',
   %%write_kp_node = 0 :: non_neg_integer() | '_'
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221096062


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.

Review Comment:
   The use of strictly monotonically increasing integer counter updates was specifically to make it easy to do calculus on the stats collected for rate of change and aggregations, so if you take a snapshot at time A and another snapshot at time B, the delta between stats divided by the window, and the delta itself is the total work done in that window.
   
   Put differently, the derivative from A to B gives the rate of change in that time window as opposed to the integral giving the total usage induced.
   
   CSRT's prupose is to get this data and make it available, but realistically you want a dedicated analysis stack like ClickHouse to do integrals on reports over time, or even `awk` over logs, but we don't preserve the logs in memory for this type of analysis. You can easily do the derivatives by way of `csrt:proc_window/3`, built on top of `recon:proc_window/3`, and exposing that over the HTTP endpoint is a great next PR on top of this work.
   
   See the HTTP docs and `csrt_query` docs for more info on the query capabilities. Much care has been taken to avoid accumulation of large quantities of `#rctx{}` entries, so this first version of CSRT cleans things out expeditiously.
   
   Another key idea of CSRT in regards to this comment is that it specifically tracks metrics we're already tracking so that you can look at the already existing node level metrics to see larger time series trends, and CSRT provides the high cardinality insight into what is actually utilizing those resources.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221111920


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```

Review Comment:
   Yeah.... I agree in general and thought about that, but chose to add it here anyways because while these are implementation details, this covers and explains the CSRT internals and fundamentals, where anyone wanting to make further additions can use this as a baseline to understand and dive in from. I don't think this CSRT.md documentation is better by the absence of this declaration of the existing fields and why they exist and what they do, so I added it in the hopes it'll help folks understand.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2337362007


##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->

Review Comment:
   > Except now that won't match against Rctxs=[] as that's valid value for the type spec [rctx()] but neither of those pattern matches are.
   
   @chewbranca The guard you have is for two different args ` is_list(Rctxs) andalso is_map(Matchers)`, I think you miss-read it as ` is_list(Rctxs) orelse is_map(Rctxs)` or something similar when you wrote you response. What Nick is proposing is
   
   ```diff
   -find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
   +find_matches(Rctxs, #{} = Matchers) when is_list(Rctxs) ->
   ```
   
   The `Rctxs` argument is not affected by the change, so the "that won't match against Rctxs=[]" comment doesn't apply.
   
   I can issue a PR with the change if you are busy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317583142


##########
src/docs/src/config/csrt.rst:
##########
@@ -0,0 +1,587 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. default-domain:: config
+.. highlight:: ini
+
+.. _config-csrt:
+
+==========================================
+Couch Stats Resource Tracker (CSRT) Config
+==========================================
+
+CSRT configuration options and overview.
+
+.. seealso::
+
+    :doc:`/csrt/index`
+
+CSRT config
+===========
+
+This section contains the top level enablement and configuration options for CSRT.
+
+.. config:section:: csrt :: CSRT Primary Configuration
+
+    .. config:option:: enable :: Enable CSRT data collection and RPC deltas
+
+        Core enablement toggle for CSRT, defaults to false. Enabling this
+        setting initiates local CSRT stats collection as well as shipping deltas
+        in RPC responses to accumulate in the coordinator.
+
+        This does *not* trigger the new RPC spawn metrics, and it does not
+        enable reporting for any of the rctx types.
+
+        .. warning::
+
+            You *MUST* have all nodes in the cluster running a CSRT aware
+            CouchDB *before* you enable it on any node, otherwise the old
+            version nodes won't know how to handle the new RPC formats
+            including an embedded Delta payload.
+
+        Top level CSRT enablement for local data collection and RPC deltas::
+
+            [csrt]
+            enable = false
+
+    .. config:option:: enable_init_p :: Enable RPC spawn metric tracking
+
+        Enablement of tracking new metric counters for different `fabric_rpc` operations
+        spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+        inducing database RPC operations within CouchDB, and these init_p metrics aim to
+        provide node level understandings of the workloads being induced by other
+        coordinator processes. This is especially relevant for databases on subsets of a
+        cluster resulting in non-uniform workloads, these metrics are tailored to
+        provide insight into what work is being spawned on each node in the cluster as a
+        function of time.
+
+        Enablement for tracking counts of spawned RPC workers::
+
+            [csrt]
+            enable_init_p = false
+
+    .. config:option:: enable_reporting :: Enable CSRT Process Lifecyle Reports
+
+        This is the primary toggle for enabling CSRT process lifetime reports
+        containing detailed information about the quantity of work induced by
+        the given request/worker/etc. This is the top level toggle for enabling
+        _any_ reporting, and there also exists
+        :config:option:`csrt/enable_rpc_reporting` to disable the reporting of
+        any individual RPC workers, leaving the coordinator responsible of
+        generating a report with the accumulated deltas.
+
+        .. note::
+
+            Note that this setting toggles whether or not to generate process
+            lifecycle reports, but no reports will be generated until logger
+            matchers have been enabled that trigger a match on CSRT contexts
+            that have surpassed the configured thresholds.
+
+        Top level toggle for whether any process lifecycle reports are generated::
+
+            [csrt]
+            enable_reporting = false
+
+    .. config:option:: enable_rpc_reporting :: Enable RPC process lifecyle reports
+
+        This enables the possibility of RPC workers generating reports. They
+        still need to hit the configured thresholds to induce a report, but
+        this will generate CSRT process lifetime reports for individual RPC
+        workers that trigger the configured logger thresholds. This allows for
+        quantifying per node resource usage when desired, as otherwise the
+        reports are at the http request level and don't provide per node stats.
+
+        The key idea here is that having RPC level CSRT process lifetime
+        reporting is incredibly useful, but can also generate large quantities
+        of data. For example, a view query on a Q=64 database will stream
+        results from 64 shard replicas, resulting in at least 64 RPC reports,
+        plus any that might have been generated from RPC workers that "lost"
+        the race for shard replica. This is very useful, but a lot of data
+        given the verbose nature of funneling it through the RSyslog reports,
+        however, the ability to write directly to something like ClickHouse or
+        another columnar store would be great.
+
+        Until there's an efficient storage mechanism to stream the results to,
+        the rsyslog entries work great and are very practical, but care must be
+        taken to not generate too much data for aggregate queries as they
+        generate at least `Qx` more report than an individual report per http
+        request from the coordinator.  This setting exists as a way to either
+        a) utilize the logger matcher configured thresholds to allow for _any_
+        rctx's to be recorded when they induce heavy operations, either
+        Coordinator or RPC worker; or b) to _only_ log workloads at the
+        coordinator level.
+
+        .. note::
+
+            This setting exists because we lack an expressive enough config
+            declaration to easily chain the matchspec constructions as
+            `ets:fun2ms/1` is a special compile time parse transform macro that
+            requires the full definition to be specified directly, it cannot
+            be interactively constructed. That said, you _can_ register matchers
+            through `remsh` with more specific and fine grained pattern matching,
+            and a more expressive system for defining matchers are being
+            explored.
+
+        .. warning::
+
+            Enabling this setting *will* generate considerably more logs! Specifically, for aggregate queries and database operations, this will generate `Q` * `N` times more logs than a singular doc request taking only `N` inreacting with a singular shard range. See the note above about this being a temporary setting during the experimental stages of CSRT.
+
+        Toggle to enable possibility of RPC process lifecycle reports::
+
+            [csrt]
+            enable_rpc_reporting = false
+
+    .. config:option:: should_truncate_reports :: truncate zero values from lifecyle reports
+
+        enables truncation of the csrt process lifetime reports to not include
+        any fields that are zero at the end of process lifetime, eg don't
+        include `js_filter=0` in the report if the request did not induce
+        javascript filtering.
+
+        this can be disabled if you really care about consistent fields in the
+        report logs, but this is a log space saving mechanism, similar to
+        disabling rpc reporting by default, as its a simple way to reduce
+        overall volume
+
+        Truncate zero values from process lifecycle reports, enabled by default:
+
+            [csrt]
+            should_truncate_reports = true
+
+    .. config:option:: query_limit :: Maximum quantity of rows to return in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_limit = 100
+
+    .. config:option:: query_cardinality_limit :: Maximum quantity of rows to allow in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_cardinality_limit = 10000
+
+.. _csrt-logger-matcher-configuration:
+
+CSRT Logger Matcher Configuration
+=================================
+
+There are currently eight builtin default logger matchers designed to make it
+easy to do filtering on heavy resource usage inducing and long running
+requests. These are designed as a simple baseline of useful matchers, declared
+in a manner amenable to `default.ini` based constructs. More expressive matcher

Review Comment:
   If we're going for a monospace code font then we'd need  a double-backtick otherwise it looks it make gives the term a slanted style. If that's what we're going for that fine:
   
   <img width="585" height="59" alt="Screenshot 2025-09-02 at 10 09 01 PM" src="https://github.com/user-attachments/assets/9187d981-f758-43dc-ad83-284bd65bd151" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317630755


##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,603 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_all_coordinators/0,
+    matcher_on_all_rpc_workers/0,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case couch_srt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = couch_srt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), couch_srt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(Msg, From, St) ->
+    {stop, {unknown_call, Msg, From}, St}.
+
+handle_cast(Msg, St) ->
+    {stop, {unknown_cast, Msg}, St}.
+
+handle_info(restart_config_listener, St) ->
+    ok = subscribe_changes(),
+    {noreply, St};
+handle_info(Msg, St) ->
+    {stop, {unknown_info, Msg}, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_all_coordinators() -> ets:match_spec().
+matcher_on_all_coordinators() ->
+    ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end).
+
+-spec matcher_on_all_rpc_workers() -> ets:match_spec().
+matcher_on_all_rpc_workers() ->
+    ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end).
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbnames_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbnames_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or

Review Comment:
   Since we used `andalso` probably makes sense to use `orelse` instead of `or`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221176094


##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->

Review Comment:
   We absolutely need `abs` here, please see my CSRT.md time documentation you commented on earlier; I specifically added that documentation because time is complicated as hell and while it's true that monotonic integer means that it's always increasing, you can _still have_ a monotonically increasing negative number, and as I specify in the documentation, this is a common occurrence!
   
   Note that https://www.erlang.org/doc/apps/erts/erlang#monotonic_time/0 returns `integer()`, _not_ `non_neg_integer()`!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236456219


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.
+
+### #rctx.rows_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([fabric_rpc, changes, processed])
+> also Tracking `couch_stats:increment_counter([fabric_rpc, view, rows_read])

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236423734


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.

Review Comment:
   The coordinator logs the total data accumulated that it was informed of, regardless of RPC reporting enablement. Again, this rpc enabled setting is an awkward pragmatic config knob for the moment, all three of the CSRT documents (the csrt/index rst docs, the config/csrt rst docs, and the couch_srt/README.md) all discuss this in further depth than the last you saw, thanks for your feedback here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236583648


##########
src/couch_srt/test/eunit/couch_srt_logger_tests.erl:
##########
@@ -0,0 +1,441 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger_tests).
+
+-import(
+    couch_srt_test_helper,
+    [
+        enable_default_logger_matchers/0,
+        rctx_gen/1,
+        rctxs/0,
+        jrctx/1
+    ]
+).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("couch_mrview/include/couch_mrview.hrl").
+-include("../../src/couch_srt.hrl").
+
+%% Use different values than default configs to ensure they're picked up
+-define(THRESHOLD_DBNAME_IO, 91).
+-define(THRESHOLD_DOCS_READ, 123).
+-define(THRESHOLD_DOCS_WRITTEN, 12).
+-define(THRESHOLD_IOQ_CALLS, 439).
+-define(THRESHOLD_ROWS_READ, 143).
+-define(THRESHOLD_CHANGES, 79).
+-define(THRESHOLD_LONG_REQS, 432).
+
+csrt_logger_reporting_works_test_() ->
+    {
+        foreach,
+        fun setup_reporting/0,
+        fun teardown_reporting/1,
+        [
+            ?TDEF_FE(t_enablement),
+            ?TDEF_FE(t_do_report),
+            ?TDEF_FE(t_do_lifetime_report),
+            ?TDEF_FE(t_do_status_report)
+        ]
+    }.
+
+csrt_logger_matchers_test_() ->
+    {
+        foreach,
+        fun setup/0,
+        fun teardown/1,
+        [
+            ?TDEF_FE(t_enablement),
+            ?TDEF_FE(t_matcher_on_dbnames_io),
+            ?TDEF_FE(t_matcher_on_docs_read),
+            ?TDEF_FE(t_matcher_on_docs_written),
+            ?TDEF_FE(t_matcher_on_rows_read),
+            ?TDEF_FE(t_matcher_on_changes_processed),
+            ?TDEF_FE(t_matcher_on_long_reqs),
+            ?TDEF_FE(t_matcher_on_ioq_calls),
+            ?TDEF_FE(t_matcher_on_nonce),
+            ?TDEF_FE(t_matcher_register_deregister)
+        ]
+    }.
+
+make_docs(Count) ->
+    lists:map(
+        fun(I) ->
+            #doc{
+                id = ?l2b("foo_" ++ integer_to_list(I)),
+                body = {[{<<"value">>, I}]}
+            }
+        end,
+        lists:seq(1, Count)
+    ).
+
+setup() ->
+    Ctx = test_util:start_couch([fabric, couch_stats, couch_srt]),
+    enable_default_logger_matchers(),
+    config:set_boolean(?CSRT, "randomize_testing", false, false),
+    config:set_boolean(?CSRT, "enable_reporting", true, false),
+    config:set_boolean(?CSRT, "enable_rpc_reporting", true, false),
+    ok = meck:new(ioq, [passthrough]),
+    ok = meck:expect(ioq, bypass, fun(_, _) -> false end),
+    DbName = ?tempdb(),
+    ok = fabric:create_db(DbName, [{q, 8}, {n, 1}]),
+    Docs = make_docs(100),
+    Opts = [],
+    {ok, _} = fabric:update_docs(DbName, Docs, Opts),
+    Method = 'GET',
+    Path = "/" ++ ?b2l(DbName) ++ "/_all_docs",
+    Nonce = couch_util:to_hex(crypto:strong_rand_bytes(5)),
+    Req = #httpd{method = Method, nonce = Nonce},
+    {_, _} = PidRef = couch_srt:create_coordinator_context(Req, Path),
+    MArgs = #mrargs{include_docs = false},
+    _Res = fabric:all_docs(DbName, [?ADMIN_CTX], fun view_cb/2, [], MArgs),
+    Rctx = load_rctx(PidRef),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "docs_read", integer_to_list(?THRESHOLD_DOCS_READ), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold",
+        "docs_written",
+        integer_to_list(?THRESHOLD_DOCS_WRITTEN),
+        false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "ioq_calls", integer_to_list(?THRESHOLD_IOQ_CALLS), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "rows_read", integer_to_list(?THRESHOLD_ROWS_READ), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold",
+        "changes_processed",
+        integer_to_list(?THRESHOLD_CHANGES),
+        false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "long_reqs", integer_to_list(?THRESHOLD_LONG_REQS), false
+    ),
+    ok = config:set("csrt_logger.dbnames_io", "foo", integer_to_list(?THRESHOLD_DBNAME_IO), false),
+    ok = config:set("csrt_logger.dbnames_io", "bar", integer_to_list(?THRESHOLD_DBNAME_IO), false),
+    ok = config:set(
+        "csrt_logger.dbnames_io", "foo/bar", integer_to_list(?THRESHOLD_DBNAME_IO), false

Review Comment:
   I fixed this in 1380b09fc as well as cleared out the use of `-import` on the test helper, like you requested elsewhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2219782432


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]

Review Comment:
   Are these clustered db name or shards? Or both?
   
   And then are they per-node or per cluster. Say foo/bar = 200 means on a node during the worker request processing it read more than that many docs or kp nodes?



##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]
+; For example:
+; foo = 100
+; _dbs = 123
+; _users = 234
+; foo/bar = 200
+
+[csrt_logger.matchers_enabled]
+;docs_read = false
+;rows_read = false
+;docs_written = false
+;long_reqs = false
+;changes_processed = false
+;ioq_calls = false
+
+[csrt_logger.matchers_threshold]

Review Comment:
   How is this different than `csrt_logger.dbnames_io` they both look like thresholds



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.

Review Comment:
   Would this be per coordinator request process or per worker request, or depends on the `enable_rpc_reporting` setting and maybe both?



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting

Review Comment:
   default.ini seems to have as `enabled`



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->

Review Comment:
   Wonder we need an abs call. If we're dealing with monotonic timestamp on the same node they'll always be non-decreasing. There are no time warps for, or are these timestamp coming from other nodes, then we can't compare them at all. 



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->
+            R
+        end
+    ).
+
+-spec matcher_on_ioq_calls(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_ioq_calls(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{ioq_calls = IOQCalls} = R) when IOQCalls >= Threshold -> R end).
+
+%%-spec matcher_for_rctx_field(Field :: rctx_field()) -> ets:match_spec().
+%%matcher_for_rctx_field() ->
+%%    #{size := Size0, fields := Fields} = csrt_entry:record_info(),
+%%    %% Subtract 1 as record_info size includes tuple record name
+%%    %% erlang:list_to_tuple([rctx | lists:duplicate(maps:get(size, csrt_entry:record_info()), '_')])
+%%    Size = Size - 1,

Review Comment:
   Debug or stray comment?



##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format
+-spec tutc(Time :: integer()) -> calendar:rfc3339_string().
+tutc(Time0) when is_integer(Time0) ->
+    Unit = millisecond,
+    Time1 = Time0 + erlang:time_offset(),
+    Time = erlang:convert_time_unit(Time1, native, Unit),
+    calendar:system_time_to_rfc3339(Time, [{unit, Unit}, {offset, "z"}]).
+
+%% Returns dt (delta time) in microseconds
+%% @equiv make_dt(A, B, microsecond)
+-spec make_dt(A, B) -> pos_integer() when
+    A :: integer(),
+    B :: integer().
+make_dt(A, B) ->
+    make_dt(A, B, microsecond).
+
+%% Returns monotonic dt (delta time) in specified time_unit()
+-spec make_dt(A, B, Unit) -> pos_integer() when
+    A :: integer(),
+    B :: integer(),
+    Unit :: erlang:time_unit().
+make_dt(A, A, _Unit) when is_integer(A) ->
+    %% Handle edge case when monotonic_time()'s are equal
+    %% Always return a non zero value so we don't divide by zero
+    %% This always returns 1, independent of unit, as that's the smallest
+    %% possible positive integer value delta.
+    1;
+make_dt(A, B, Unit) when is_integer(A) andalso is_integer(B) andalso B > A ->
+    case erlang:convert_time_unit(abs(B - A), native, Unit) of
+        Delta when Delta > 0 ->
+            Delta;
+        _ ->
+            %% Handle case where Delta is smaller than a whole Unit, eg:
+            %% Unit = millisecond,
+            %% (node1@127.0.0.1)2> erlang:convert_time_unit(423, native, Unit).
+            %% 0
+            1
+    end.
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, undefined) ->
+    T;
+add_delta(T, Delta) when is_map(Delta) ->
+    add_delta_int(T, {delta, Delta}).
+
+-spec add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+add_delta_int(T, {delta, _} = Delta) ->
+    {T, Delta}.
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta({Msg, {delta, Delta}}) ->
+    {Msg, Delta};
+extract_delta(Msg) ->
+    {Msg, undefined}.
+
+-spec get_delta(PidRef :: maybe_pid_ref()) -> tagged_delta().
+get_delta(PidRef) ->
+    {delta, make_delta(PidRef)}.
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            maybe_add_delta_int(T, get_delta(get_pid_ref()))
+    end.
+
+%% Allow for externally provided Delta in error handling scenarios
+%% eg in cases like rexi_server:notify_caller/3
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, undefined) ->
+    T;
+maybe_add_delta(T, Delta0) when is_map(Delta0) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            Delta = {delta, Delta0},
+            maybe_add_delta_int(T, Delta)
+    end.
+
+-spec maybe_add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+maybe_add_delta_int(T, {delta, undefined}) ->
+    T;
+maybe_add_delta_int(T, {delta, _} = Delta) ->
+    add_delta_int(T, Delta).
+
+-spec make_delta(PidRef :: maybe_pid_ref()) -> maybe_delta().
+make_delta(undefined) ->
+    undefined;
+make_delta(PidRef) ->
+    TA = get_delta_a(),
+    TB = csrt_server:get_resource(PidRef),
+    Delta = rctx_delta(TA, TB),
+    put_delta_a(TB),
+    Delta.
+
+-spec rctx_delta(TA :: Rctx, TB :: Rctx) -> map().
+rctx_delta(#rctx{} = TA, #rctx{} = TB) ->
+    Delta = #{
+        docs_read => TB#rctx.docs_read - TA#rctx.docs_read,
+        docs_written => TB#rctx.docs_written - TA#rctx.docs_written,
+        js_filter => TB#rctx.js_filter - TA#rctx.js_filter,
+        js_filtered_docs => TB#rctx.js_filtered_docs - TA#rctx.js_filtered_docs,
+        rows_read => TB#rctx.rows_read - TA#rctx.rows_read,
+        changes_returned => TB#rctx.changes_returned - TA#rctx.changes_returned,
+        get_kp_node => TB#rctx.get_kp_node - TA#rctx.get_kp_node,
+        get_kv_node => TB#rctx.get_kv_node - TA#rctx.get_kv_node,
+        db_open => TB#rctx.db_open - TA#rctx.db_open,
+        ioq_calls => TB#rctx.ioq_calls - TA#rctx.ioq_calls,
+        %% "Example to extend CSRT"
+        %% write_kp_node => TB#rctx.write_kp_node - TA#rctx.write_kp_node,
+        %% write_kv_node => TB#rctx.write_kv_node - TA#rctx.write_kv_node,
+        dt => make_dt(TA#rctx.updated_at, TB#rctx.updated_at)
+    },
+    %% TODO: reevaluate this decision
+    %% Only return non zero (and also positive) delta fields
+    %% NOTE: this can result in Delta's of the form #{dt => 1}
+    maps:filter(fun(_K, V) -> V > 0 end, Delta);
+rctx_delta(_, _) ->
+    undefined.
+
+-spec get_delta_a() -> maybe_rctx().
+get_delta_a() ->
+    erlang:get(?DELTA_TA).
+
+-spec put_delta_a(TA :: rctx()) -> maybe_rctx().
+put_delta_a(TA) ->
+    erlang:put(?DELTA_TA, TA).
+
+-spec get_updated_at() -> maybe_integer().
+get_updated_at() ->
+    erlang:get(?LAST_UPDATED).
+
+-spec put_updated_at(Updated :: rctx() | integer()) -> maybe_integer().
+put_updated_at(#rctx{updated_at = Updated}) ->
+    put_updated_at(Updated);
+put_updated_at(Updated) when is_integer(Updated) ->
+    erlang:put(?LAST_UPDATED, Updated).
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    get(?PID_REF).
+
+-spec get_pid_ref(Rctx :: rctx()) -> maybe_pid_ref().
+get_pid_ref(#rctx{pid_ref = PidRef}) ->
+    PidRef;
+get_pid_ref(_) ->
+    undefined.
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    erlang:put(?PID_REF, PidRef),
+    PidRef.
+
+%% @equiv set_fabric_init_p(Func, Enabled, true).
+-spec set_fabric_init_p(Func :: atom(), Enabled :: boolean()) -> ok.
+set_fabric_init_p(Func, Enabled) ->
+    set_fabric_init_p(Func, Enabled, true).
+
+%% Expose Persist for use in test cases outside this module
+-spec set_fabric_init_p(Func, Enabled, Persist) -> ok when
+    Func :: atom(), Enabled :: boolean(), Persist :: boolean().
+set_fabric_init_p(Func, Enabled, Persist) ->
+    Key = fabric_conf_key(Func),
+    ok = config:set_boolean(?CSRT_INIT_P, Key, Enabled, Persist).
+
+-spec fabric_conf_key(Key :: atom()) -> string().
+fabric_conf_key(Key) ->
+    %% Double underscore to separate Mod and Func
+    "fabric_rpc__" ++ atom_to_list(Key).

Review Comment:
   Use `:`? Double-underscore is an unusual convention of `module:function` references



##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true

Review Comment:
   Usually we have commented out values here from the code. Not sure if we should make them enabled or disabled. When would a user chose to enable/disable them?
   
   Also use the more common convention of `m:f` of double-underscore https://www.erlang.org/doc/system/funs.html#syntax-of-funs. The typical syntax is `fun m:f/a` but we don't care about the arity I think



##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true

Review Comment:
   In the default ini we normally don't have actual value, just default from the code. 



##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true

Review Comment:
    Docs seem to indicate this is disabled



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.

Review Comment:
   Would any stats usage from a long running request be visible while the request is running or just when it finishes. Say, while the request is running for an hour, we would get delta reports and see what's consuming things but we only get the logged report at the end of the request?



##########
src/couch_stats/src/csrt_entry.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_entry).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-export([
+    value/2,
+    key/1,
+    from_map/1,
+    record_info/0
+]).
+
+%% JSON Conversion API
+-export([
+    convert_type/1,
+    convert_pidref/1,
+    convert_pid/1,
+    convert_ref/1,
+    convert_string/1,
+    to_json/1
+]).
+
+-spec value(rctx_field(), #rctx{}) -> any().
+
+value(pid_ref, #rctx{pid_ref = Val}) -> Val;
+value(nonce, #rctx{nonce = Val}) -> Val;
+value(type, #rctx{type = Val}) -> convert_type(Val);
+value(dbname, #rctx{dbname = Val}) -> Val;
+value(username, #rctx{username = Val}) -> Val;
+value(db_open, #rctx{db_open = Val}) -> Val;
+value(docs_read, #rctx{docs_read = Val}) -> Val;
+value(docs_written, #rctx{docs_written = Val}) -> Val;
+value(rows_read, #rctx{rows_read = Val}) -> Val;
+value(changes_returned, #rctx{changes_returned = Val}) -> Val;
+value(ioq_calls, #rctx{ioq_calls = Val}) -> Val;
+value(js_filter, #rctx{js_filter = Val}) -> Val;
+value(js_filtered_docs, #rctx{js_filtered_docs = Val}) -> Val;
+value(get_kv_node, #rctx{get_kv_node = Val}) -> Val;
+value(get_kp_node, #rctx{get_kp_node = Val}) -> Val;
+value(started_at, #rctx{started_at = Val}) -> Val;
+value(updated_at, #rctx{updated_at = Val}) -> Val.
+
+-spec key(BinKey :: binary() | string() | atom()) ->
+    Key ::
+        rctx_field()
+        | {error, Reason :: any()}.
+
+key(Key) when is_atom(Key) ->
+    key_from_atom(Key);
+key(Key) when is_binary(Key) ->
+    key_from_binary(Key);
+key(Key) when is_list(Key) ->
+    case key_from_binary(list_to_binary(Key)) of
+        {error, {invalid_key, _Key}} ->
+            {error, {invalid_key, Key}};
+        Res ->
+            Res
+    end;
+key(Other) ->
+    key_error(Other).
+
+key_from_atom(pid_ref) -> pid_ref;
+key_from_atom(nonce) -> nonce;
+key_from_atom(type) -> type;
+key_from_atom(dbname) -> dbname;
+key_from_atom(username) -> username;
+key_from_atom(db_open) -> db_open;
+key_from_atom(docs_read) -> docs_read;
+key_from_atom(rows_read) -> rows_read;
+key_from_atom(changes_returned) -> changes_returned;
+key_from_atom(ioq_calls) -> ioq_calls;
+key_from_atom(js_filter) -> js_filter;
+key_from_atom(js_filtered_docs) -> js_filtered_docs;
+key_from_atom(get_kv_node) -> get_kv_node;
+key_from_atom(get_kp_node) -> get_kp_node;

Review Comment:
   These all seem to be exactly the same except for key_error case. This is basically to validate the key?
   
   Wonder if adding a of fields in the define and just do lists:member or a set and check for membership would make it less verbose.



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker

Review Comment:
   reprots -> reports



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.

Review Comment:
   > The key idea here is that having RPC level CSRT process lifetime reporting is
   incredibly useful, 
   
   Yeah that seems useful. Or more in general a user might be nice to know per node which requests (db name, users) are performing more than `Threshold` number of a particular IO requests per time unit (minute, second, etc).  Would we be able to get that using CSRT? 



##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format

Review Comment:
   Maybe a clarification needed, the `Time0` argument we get is not in milliseconds but in `native` units right? And it's the monotonic timestamp from the same node? So we add the offset + Time0 in native then convert wan to convert it to rfc3339 with 3 digits millisecond precision.
   
   I think something like this one-liner could work then:
   
   ```erlang
   calendar:system_time_to_rfc3339(Time0 + erlang:time_offset(), [{unit, native}, {offset, "z"}]).
   ```



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).

Review Comment:
   Stray comment?



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).

Review Comment:
   Noticed the deregister calls doesn't have an infinity timeout while the register one does



##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   Avoid imports if possible. We use them in a few places in http code but unless they make the code too verbose.
   
   For instance now we have a mix of full references like `csrt:rpc_run()` and `chttpd:verify_is_server_admin()` vs imported ones like `query()` and it seems kind of arbitrary



##########
src/couch_stats/src/csrt_entry.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_entry).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-export([
+    value/2,
+    key/1,
+    from_map/1,
+    record_info/0
+]).
+
+%% JSON Conversion API
+-export([
+    convert_type/1,
+    convert_pidref/1,
+    convert_pid/1,
+    convert_ref/1,
+    convert_string/1,
+    to_json/1
+]).
+
+-spec value(rctx_field(), #rctx{}) -> any().
+
+value(pid_ref, #rctx{pid_ref = Val}) -> Val;
+value(nonce, #rctx{nonce = Val}) -> Val;
+value(type, #rctx{type = Val}) -> convert_type(Val);
+value(dbname, #rctx{dbname = Val}) -> Val;
+value(username, #rctx{username = Val}) -> Val;
+value(db_open, #rctx{db_open = Val}) -> Val;
+value(docs_read, #rctx{docs_read = Val}) -> Val;
+value(docs_written, #rctx{docs_written = Val}) -> Val;
+value(rows_read, #rctx{rows_read = Val}) -> Val;
+value(changes_returned, #rctx{changes_returned = Val}) -> Val;
+value(ioq_calls, #rctx{ioq_calls = Val}) -> Val;
+value(js_filter, #rctx{js_filter = Val}) -> Val;
+value(js_filtered_docs, #rctx{js_filtered_docs = Val}) -> Val;
+value(get_kv_node, #rctx{get_kv_node = Val}) -> Val;
+value(get_kp_node, #rctx{get_kp_node = Val}) -> Val;
+value(started_at, #rctx{started_at = Val}) -> Val;
+value(updated_at, #rctx{updated_at = Val}) -> Val.
+
+-spec key(BinKey :: binary() | string() | atom()) ->
+    Key ::
+        rctx_field()
+        | {error, Reason :: any()}.
+
+key(Key) when is_atom(Key) ->
+    key_from_atom(Key);
+key(Key) when is_binary(Key) ->
+    key_from_binary(Key);
+key(Key) when is_list(Key) ->
+    case key_from_binary(list_to_binary(Key)) of
+        {error, {invalid_key, _Key}} ->
+            {error, {invalid_key, Key}};
+        Res ->
+            Res
+    end;
+key(Other) ->
+    key_error(Other).
+
+key_from_atom(pid_ref) -> pid_ref;
+key_from_atom(nonce) -> nonce;
+key_from_atom(type) -> type;
+key_from_atom(dbname) -> dbname;
+key_from_atom(username) -> username;
+key_from_atom(db_open) -> db_open;
+key_from_atom(docs_read) -> docs_read;
+key_from_atom(rows_read) -> rows_read;
+key_from_atom(changes_returned) -> changes_returned;
+key_from_atom(ioq_calls) -> ioq_calls;
+key_from_atom(js_filter) -> js_filter;
+key_from_atom(js_filtered_docs) -> js_filtered_docs;
+key_from_atom(get_kv_node) -> get_kv_node;
+key_from_atom(get_kp_node) -> get_kp_node;
+key_from_atom(Other) -> key_error(Other).
+
+key_from_binary(<<"pid_ref">>) -> pid_ref;
+key_from_binary(<<"nonce">>) -> nonce;
+key_from_binary(<<"type">>) -> type;
+key_from_binary(<<"dbname">>) -> dbname;
+key_from_binary(<<"username">>) -> username;
+key_from_binary(<<"db_open">>) -> db_open;
+key_from_binary(<<"docs_read">>) -> docs_read;
+key_from_binary(<<"rows_read">>) -> rows_read;
+key_from_binary(<<"changes_returned">>) -> changes_returned;
+key_from_binary(<<"ioq_calls">>) -> ioq_calls;
+key_from_binary(<<"js_filter">>) -> js_filter;
+key_from_binary(<<"js_filtered_docs">>) -> js_filtered_docs;
+key_from_binary(<<"get_kv_node">>) -> get_kv_node;
+key_from_binary(<<"get_kp_node">>) -> get_kp_node;
+key_from_binary(Other) -> key_error(Other).

Review Comment:
   This seems like a binary_to_atom + validation. Maybe try a binary to exiting atom and if it passes also do a key_form atom if that has a specific field validation?



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```

Review Comment:
   Minute nit: avoid duplicating code in the docs. A reference to module or file name is good enough.



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `csrt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `csrt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `csrt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = csrt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = csrt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `csrt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `csrt:set_context_username/{1,2}`.

Review Comment:
   The descriptions of individual struct fields would probably go better in the code a comments or specs. Having them in the docs risks them getting out of sync with time. 



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `csrt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```

Review Comment:
   It's a good note but probably don't need to quote the docs in our docs. It feels a bit too low level. Maybe a comment link in the function on what `erlang:monotonic_time()` return is enough.



##########
src/couch_stats/src/csrt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case csrt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling csrt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case csrt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = csrt_util:tnow(),
+            csrt_util:put_updated_at(Now),
+            UpdatedInc = csrt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(_, _From, State) ->
+    {reply, ok, State}.

Review Comment:
   It's probably be better to stop / crash on an unknown call and not paper over it.



##########
src/couch_stats/src/csrt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case csrt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling csrt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case csrt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = csrt_util:tnow(),
+            csrt_util:put_updated_at(Now),
+            UpdatedInc = csrt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.

Review Comment:
   What does the `0}` do? We just want to ignore the casts  I think. Even cleaner might be to crash 



##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format
+-spec tutc(Time :: integer()) -> calendar:rfc3339_string().
+tutc(Time0) when is_integer(Time0) ->
+    Unit = millisecond,
+    Time1 = Time0 + erlang:time_offset(),
+    Time = erlang:convert_time_unit(Time1, native, Unit),
+    calendar:system_time_to_rfc3339(Time, [{unit, Unit}, {offset, "z"}]).
+
+%% Returns dt (delta time) in microseconds
+%% @equiv make_dt(A, B, microsecond)
+-spec make_dt(A, B) -> pos_integer() when
+    A :: integer(),
+    B :: integer().
+make_dt(A, B) ->
+    make_dt(A, B, microsecond).
+
+%% Returns monotonic dt (delta time) in specified time_unit()
+-spec make_dt(A, B, Unit) -> pos_integer() when
+    A :: integer(),
+    B :: integer(),
+    Unit :: erlang:time_unit().
+make_dt(A, A, _Unit) when is_integer(A) ->
+    %% Handle edge case when monotonic_time()'s are equal
+    %% Always return a non zero value so we don't divide by zero
+    %% This always returns 1, independent of unit, as that's the smallest
+    %% possible positive integer value delta.
+    1;
+make_dt(A, B, Unit) when is_integer(A) andalso is_integer(B) andalso B > A ->
+    case erlang:convert_time_unit(abs(B - A), native, Unit) of

Review Comment:
   If we assert B > A then we probably don't need the abs call?



##########
src/couch_stats/src/couch_stats.erl:
##########
@@ -49,6 +54,10 @@ increment_counter(Name) ->
 
 -spec increment_counter(any(), pos_integer()) -> response().
 increment_counter(Name, Value) ->
+    %% Should maybe_track_local happen before or after notify?
+    %% If after, only currently tracked metrics declared in the app's
+    %% stats_description.cfg will be trackable locally. Pros/cons.
+    ok = maybe_track_local_counter(Name, Value),

Review Comment:
   This seems to be the only integration point with the couch_stats application. And even her csrt could be turned off or crashed and we'd still be going. There is a separate supervisor, separate utils module, separate .h files. So I think it makes sense to have it in a separate application. Then it can also have a more conventional names like `couch_resource_tracker` or `couch_rt` if that's that's too long. 
   
   



##########
src/rexi/src/rexi_monitor.erl:
##########
@@ -35,6 +35,7 @@ start(Procs) ->
 %% messages from our mailbox.
 -spec stop(pid()) -> ok.
 stop(MonitoringPid) ->
+    unlink(MonitoringPid),

Review Comment:
   Why do we need to unlink?



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->
+            R
+        end
+    ).
+
+-spec matcher_on_ioq_calls(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_ioq_calls(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{ioq_calls = IOQCalls} = R) when IOQCalls >= Threshold -> R end).
+
+%%-spec matcher_for_rctx_field(Field :: rctx_field()) -> ets:match_spec().
+%%matcher_for_rctx_field() ->
+%%    #{size := Size0, fields := Fields} = csrt_entry:record_info(),
+%%    %% Subtract 1 as record_info size includes tuple record name
+%%    %% erlang:list_to_tuple([rctx | lists:duplicate(maps:get(size, csrt_entry:record_info()), '_')])
+%%    Size = Size - 1,
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> matcher() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    #{field_idx := FieldIdx} = csrt_entry:record_info(),
+    RctxMatch0 = #rctx{_ = '_'},
+    RctxMatch1 = setelement(maps:get(pid_ref, FieldIdx) + 1, RctxMatch0, '$1'),
+    RctxMatch = setelement(maps:get(AttrName, FieldIdx) + 1, RctxMatch1, '$2'),
+    MatchSpec = [{RctxMatch, [], [{{'$1', '$2'}}]}],
+    {MatchSpec, ets:match_spec_compile(MatchSpec)}.
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> list() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    {MatchSpec, _CompMatch} = pid_ref_matchspec(AttrName),
+    %% Base fields at least an empty list, but we could add more info here.
+    %% The recon typespec is an improper list of the form:
+    %%Base = [Name | [{current_function, mfa()} | {initial_call, mfa()}]],
+    Base = [],
+    [{PidRef, Val, Base} || {PidRef, Val} <- ets:select(?CSRT_ETS, MatchSpec)].
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+
+-spec add_matcher(Name, MSpec, Matchers) ->
+    {ok, matchers()} | {error, {invalid_ms, string(), ets:match_spec()}}
+when
+    Name :: string(), MSpec :: ets:match_spec(), Matchers :: matchers().
+add_matcher(Name, MSpec, Matchers) ->
+    try ets:match_spec_compile(MSpec) of
+        CompMSpec ->
+            %% TODO: handle already registered name case
+            Matchers1 = maps:put(Name, {MSpec, CompMSpec}, Matchers),

Review Comment:
   Nit: a map syntax based update is usually more common in the code-base than map:put, we don't do a lot of map:puts



##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->
+            R
+        end
+    ).
+
+-spec matcher_on_ioq_calls(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_ioq_calls(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{ioq_calls = IOQCalls} = R) when IOQCalls >= Threshold -> R end).
+
+%%-spec matcher_for_rctx_field(Field :: rctx_field()) -> ets:match_spec().
+%%matcher_for_rctx_field() ->
+%%    #{size := Size0, fields := Fields} = csrt_entry:record_info(),
+%%    %% Subtract 1 as record_info size includes tuple record name
+%%    %% erlang:list_to_tuple([rctx | lists:duplicate(maps:get(size, csrt_entry:record_info()), '_')])
+%%    Size = Size - 1,
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> matcher() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    #{field_idx := FieldIdx} = csrt_entry:record_info(),
+    RctxMatch0 = #rctx{_ = '_'},
+    RctxMatch1 = setelement(maps:get(pid_ref, FieldIdx) + 1, RctxMatch0, '$1'),
+    RctxMatch = setelement(maps:get(AttrName, FieldIdx) + 1, RctxMatch1, '$2'),
+    MatchSpec = [{RctxMatch, [], [{{'$1', '$2'}}]}],
+    {MatchSpec, ets:match_spec_compile(MatchSpec)}.
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> list() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    {MatchSpec, _CompMatch} = pid_ref_matchspec(AttrName),
+    %% Base fields at least an empty list, but we could add more info here.
+    %% The recon typespec is an improper list of the form:
+    %%Base = [Name | [{current_function, mfa()} | {initial_call, mfa()}]],
+    Base = [],
+    [{PidRef, Val, Base} || {PidRef, Val} <- ets:select(?CSRT_ETS, MatchSpec)].
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+
+-spec add_matcher(Name, MSpec, Matchers) ->
+    {ok, matchers()} | {error, {invalid_ms, string(), ets:match_spec()}}
+when
+    Name :: string(), MSpec :: ets:match_spec(), Matchers :: matchers().
+add_matcher(Name, MSpec, Matchers) ->
+    try ets:match_spec_compile(MSpec) of
+        CompMSpec ->
+            %% TODO: handle already registered name case
+            Matchers1 = maps:put(Name, {MSpec, CompMSpec}, Matchers),
+            {ok, Matchers1}
+    catch
+        error:badarg ->
+            {error, {invalid_ms, Name, MSpec}}
+    end.
+
+-spec set_matchers_term(Matchers :: matchers()) -> ok.
+set_matchers_term(Matchers) when is_map(Matchers) ->
+    persistent_term:put(?MATCHERS_KEY, Matchers).
+
+-spec initialize_matchers(RegisteredMatchers :: map()) -> ok.
+initialize_matchers(RegisteredMatchers) when is_map(RegisteredMatchers) ->
+    %% Standard matchers to conditionally enable
+    DefaultMatchers = [
+        {docs_read, fun matcher_on_docs_read/1, 1000},
+        {rows_read, fun matcher_on_rows_read/1, 1000},
+        {docs_written, fun matcher_on_docs_written/1, 500},
+        %% long_reqs Threshold in milliseconds
+        {long_reqs, fun matcher_on_long_reqs/1, 60000},
+        {changes_processed, fun matcher_on_changes_processed/1, 1000},
+        {ioq_calls, fun matcher_on_ioq_calls/1, 10000}
+    ],
+
+    %% Add enabled Matchers for standard matchers
+    Matchers = lists:foldl(
+        fun({Name0, MatchGenFunc, Threshold0}, Matchers0) when is_atom(Name0) ->
+            Name = atom_to_list(Name0),
+            case matcher_enabled(Name) of
+                true ->
+                    %% Wrap in a try-catch to handle MatcherGen errors
+                    try
+                        Threshold = matcher_threshold(Name, Threshold0),
+                        case add_matcher(Name, MatchGenFunc(Threshold), Matchers0) of
+                            {ok, Matchers1} ->
+                                Matchers1;
+                            {error, {invalid_ms, NameE, MSpecE}} ->
+                                couch_log:warning("[~p] Failed to initialize matcher[~p]: ~p", [
+                                    ?MODULE, NameE, MSpecE
+                                ]),
+                                Matchers0
+                        end
+                    catch
+                        _:_ ->
+                            Matchers0
+                    end;
+                false ->
+                    couch_log:warning("[~p] Failed to initialize matcher: ~p", [
+                        ?MODULE, Name
+                    ]),
+                    Matchers0
+            end
+        end,
+        #{},
+        DefaultMatchers
+    ),
+
+    %% Add additional dbname_io matchers
+    Matchers1 = lists:foldl(
+        fun({Dbname, Value}, Matchers0) ->
+            try list_to_integer(Value) of
+                Threshold when Threshold > 0 ->
+                    Name = "dbname_io__" ++ Dbname ++ "__" ++ Value,
+                    DbnameB = list_to_binary(Dbname),
+                    MSpec = matcher_on_dbname_io_threshold(DbnameB, Threshold),
+                    case add_matcher(Name, MSpec, Matchers0) of
+                        {ok, Matchers1} ->
+                            Matchers1;
+                        {error, {invalid_ms, NameE, MSpecE}} ->
+                            couch_log:warning("[~p] Failed to initialize matcher[~p]: ~p", [
+                                ?MODULE, NameE, MSpecE
+                            ]),
+                            Matchers0
+                    end;
+                _ ->
+                    Matchers0
+            catch
+                error:badarg ->
+                    couch_log:warning("[~p] Failed to initialize dbname io matcher on[~p]: ~p", [
+                        ?MODULE, Dbname, Value
+                    ]),
+                    Matchers0
+            end
+        end,
+        Matchers,
+        config:get(?CSRT_MATCHERS_DBNAMES)
+    ),
+
+    %% Finally, merge in the dynamically registered matchers, with priority
+    Matchers2 = maps:merge(Matchers1, RegisteredMatchers),
+
+    couch_log:notice("Initialized ~p CSRT Logger matchers", [maps:size(Matchers2)]),
+    set_matchers_term(Matchers2),
+    ok.
+
+-spec matcher_enabled(Name :: string()) -> boolean().
+matcher_enabled(Name) when is_list(Name) ->
+    %% TODO: fix
+    %% config:get_boolean(?CSRT_MATCHERS_ENABLED, Name, false).

Review Comment:
   Left-over todo item?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2219920633


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   Avoid imports if possible. We use them in a few places in http code but they're not a great of a feature.
   
   For instance now we have a mix of full references like `csrt:rpc_run()` and `chttpd:verify_is_server_admin()` vs imported ones like `query()` and it seems kind of arbitrary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226679376


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need

Review Comment:
   The thresholds here are per node per worker if enable_rpc_reporting = true? If enable_rpc_reporting = false do those thresholds not take effect or they work only on the coordinator side the?



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.

Review Comment:
   We do ship delta stats across rexi to the coordinators are those not the node stats? Or we just see them at the end of the request when/if a report is generated at the end from the coordinator



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.

Review Comment:
   If the rows are truncated is there an indication in the response? If this meant to be used by cluster operators wonder if the extra safety is even needed, it would be one off request and the'd probably want to wait to get all the rows than forcing them to build an iteration / bookmarking thing.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").

Review Comment:
   Avoid using macros in the docs, just plain section and key - value string. Users then can read just one section of the docs and don't have to scroll through to find what the CSRT_MATCHERS_THRESHOLD macro is.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```

Review Comment:
   What can the user do with this API? Maybe add something like a "you might want to call these if / when ..." kind of a section. If it's mostly an internal API wouldn't make sense for users to call it, maybe skip mentioning, add a comment in the code near them if there is something interesting or surprising about them.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.

Review Comment:
   Hmm, maybe we should make it a real metrics: fabric_doc_written or something? Would it tell us anything interesting or would be pretty much identical to the one db updater it would be just a waste...



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.

Review Comment:
   That's why I was thinking maybe having a per/node per/time window summary mode might help to avoid the logs being overloaded and still get the rpc-level data. Kind of a middle ground or that works could be just the default and not have two separate coordinator only vs coordinator + enable_rpc_reporting and have less choices for users to pick from.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.

Review Comment:
   Maybe it's better to skip mentioning it? Users will likely either toggle or not toggle csrt reporting on a cluster and likely won't run eunit tests. For eunit tests a note for devs in the tests module is good enough.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a

Review Comment:
   Users will likely not read the macros from the code so better to use string. Config settings should also probably also go into the main docs area for config values. There is some special syntax there to cross link them and such and some default formatting applied and such.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.

Review Comment:
   How does the mechanism work with the coordinator level reporting vs rpc worker level? If we have coordinator level reports would `docs_reads` get the funneled docs_read from rpc worker via the "delta" rexi bit or it's just the rows sent to the client?



##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   Is this the erlang `string()` type or own locally defined version. I don't remember types very well but looking it up I see `-type string() :: [[char]]` which a list of character so if we emit it as with jiffy won't it be encoded as an array?
   
   ```
   jiffy:encode(#{value => [$a, $b, $c]}).
   <<"{\"value\":[97,98,99]}">>
   ```
   



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.

Review Comment:
   How would this work with the  rpc worker reporting? There we'd look at the process lifetime of the worker and then emit a report and then likely the coordinator will do the same thing? Or this is triggered just on the coordinator process?



##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->

Review Comment:
   Suggestion: up to you but often `#{} = ` works just as well instead of `is_map/1` and avoid an extra `andalso` guard



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.

Review Comment:
   Like the other such config kv values should also go into the main config section in the docs to be properly formatted and displayed there. Also like in the other comment, avoid using code macros. Developers who understand what erlang macros are can probably find them in the code as just as well and we'll avoid them getting out of sync with time.



##########
src/couch_srt/test/eunit/couch_srt_logger_tests.erl:
##########
@@ -0,0 +1,441 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger_tests).
+
+-import(
+    couch_srt_test_helper,
+    [
+        enable_default_logger_matchers/0,
+        rctx_gen/1,
+        rctxs/0,
+        jrctx/1
+    ]
+).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("couch_mrview/include/couch_mrview.hrl").
+-include("../../src/couch_srt.hrl").
+
+%% Use different values than default configs to ensure they're picked up
+-define(THRESHOLD_DBNAME_IO, 91).
+-define(THRESHOLD_DOCS_READ, 123).
+-define(THRESHOLD_DOCS_WRITTEN, 12).
+-define(THRESHOLD_IOQ_CALLS, 439).
+-define(THRESHOLD_ROWS_READ, 143).
+-define(THRESHOLD_CHANGES, 79).
+-define(THRESHOLD_LONG_REQS, 432).
+
+csrt_logger_reporting_works_test_() ->
+    {
+        foreach,
+        fun setup_reporting/0,
+        fun teardown_reporting/1,
+        [
+            ?TDEF_FE(t_enablement),
+            ?TDEF_FE(t_do_report),
+            ?TDEF_FE(t_do_lifetime_report),
+            ?TDEF_FE(t_do_status_report)
+        ]
+    }.
+
+csrt_logger_matchers_test_() ->
+    {
+        foreach,
+        fun setup/0,
+        fun teardown/1,
+        [
+            ?TDEF_FE(t_enablement),
+            ?TDEF_FE(t_matcher_on_dbnames_io),
+            ?TDEF_FE(t_matcher_on_docs_read),
+            ?TDEF_FE(t_matcher_on_docs_written),
+            ?TDEF_FE(t_matcher_on_rows_read),
+            ?TDEF_FE(t_matcher_on_changes_processed),
+            ?TDEF_FE(t_matcher_on_long_reqs),
+            ?TDEF_FE(t_matcher_on_ioq_calls),
+            ?TDEF_FE(t_matcher_on_nonce),
+            ?TDEF_FE(t_matcher_register_deregister)
+        ]
+    }.
+
+make_docs(Count) ->
+    lists:map(
+        fun(I) ->
+            #doc{
+                id = ?l2b("foo_" ++ integer_to_list(I)),
+                body = {[{<<"value">>, I}]}
+            }
+        end,
+        lists:seq(1, Count)
+    ).
+
+setup() ->
+    Ctx = test_util:start_couch([fabric, couch_stats, couch_srt]),
+    enable_default_logger_matchers(),
+    config:set_boolean(?CSRT, "randomize_testing", false, false),
+    config:set_boolean(?CSRT, "enable_reporting", true, false),
+    config:set_boolean(?CSRT, "enable_rpc_reporting", true, false),
+    ok = meck:new(ioq, [passthrough]),
+    ok = meck:expect(ioq, bypass, fun(_, _) -> false end),
+    DbName = ?tempdb(),
+    ok = fabric:create_db(DbName, [{q, 8}, {n, 1}]),
+    Docs = make_docs(100),
+    Opts = [],
+    {ok, _} = fabric:update_docs(DbName, Docs, Opts),
+    Method = 'GET',
+    Path = "/" ++ ?b2l(DbName) ++ "/_all_docs",
+    Nonce = couch_util:to_hex(crypto:strong_rand_bytes(5)),
+    Req = #httpd{method = Method, nonce = Nonce},
+    {_, _} = PidRef = couch_srt:create_coordinator_context(Req, Path),
+    MArgs = #mrargs{include_docs = false},
+    _Res = fabric:all_docs(DbName, [?ADMIN_CTX], fun view_cb/2, [], MArgs),
+    Rctx = load_rctx(PidRef),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "docs_read", integer_to_list(?THRESHOLD_DOCS_READ), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold",
+        "docs_written",
+        integer_to_list(?THRESHOLD_DOCS_WRITTEN),
+        false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "ioq_calls", integer_to_list(?THRESHOLD_IOQ_CALLS), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "rows_read", integer_to_list(?THRESHOLD_ROWS_READ), false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold",
+        "changes_processed",
+        integer_to_list(?THRESHOLD_CHANGES),
+        false
+    ),
+    ok = config:set(
+        "csrt_logger.matchers_threshold", "long_reqs", integer_to_list(?THRESHOLD_LONG_REQS), false
+    ),
+    ok = config:set("csrt_logger.dbnames_io", "foo", integer_to_list(?THRESHOLD_DBNAME_IO), false),
+    ok = config:set("csrt_logger.dbnames_io", "bar", integer_to_list(?THRESHOLD_DBNAME_IO), false),
+    ok = config:set(
+        "csrt_logger.dbnames_io", "foo/bar", integer_to_list(?THRESHOLD_DBNAME_IO), false

Review Comment:
   Minor DRY suggestion: maybe create local config_set function that does the integer_to_list  and persist = false to reduce verbosity a bit seeing as we're having to wrap these into 6 different lines and there is a whole bunch of them repeating? Up to you, if you think it would look better



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```

Review Comment:
   Would these be useful to call from remsh? Maybe  double-check that csrt is enabled? If the user is not meant to call them when setting up or operating csrt they can just summarized in short blurb about csrt module how "it has these interesting functions you may want to call, and a bunch of other but see the module comments, etc".
   
    In general avoid copy-pasting code section in the docs as it will get out of sync eventually and lead to confusion.



##########
src/couch_srt/src/couch_srt.erl:
##########
@@ -0,0 +1,952 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+%% PidRef API
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+
+%% Context Lifecycle API
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+
+%% Public API
+-export([
+    maybe_track_rexi_init_p/1,
+    maybe_track_local_counter/2,
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+
+%% Stats Collection API
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+
+%% RPC API
+-export([
+    rpc_run/1,
+    rpc_unsafe_run/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-opaque query() :: couch_srt_query:query().
+-opaque query_expression() :: couch_srt_query:query_expression().
+-opaque query_option() :: couch_srt_query:query_option().
+
+%%
+%% RPC Operations
+%%
+
+-spec rpc_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, run, [Query])).
+
+-spec rpc_unsafe_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_unsafe_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, unsafe_run, [Query])).
+
+merge_results(Nodes, Resp) ->
+    %% The result of erpc:multicall is returned as a list where the result from each
+    %% node is placed at the same position as the node name is placed in Nodes.
+    %% That is why we can use `lists:zip/2` here.
+    lists:map(fun format_response/1, lists:zip(Nodes, Resp)).
+
+format_response({Node, {ok, {ok, Result}}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {ok, {error, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {ok, Result}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {error, {erpc, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {Tag, _}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Tag]
+    }.
+
+%%
+%% PidRef Operations
+%%
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    couch_srt_util:get_pid_ref().
+
+-spec get_pid_ref(Rctx :: rctx()) -> pid_ref().
+get_pid_ref(Rctx) ->
+    couch_srt_util:get_pid_ref(Rctx).
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    couch_srt_util:set_pid_ref(PidRef).
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    couch_srt_server:create_pid_ref().
+
+-spec destroy_pid_ref() -> maybe_pid_ref().
+destroy_pid_ref() ->
+    destroy_pid_ref(get_pid_ref()).
+
+%%destroy_pid_ref(undefined) ->
+%%    undefined;
+-spec destroy_pid_ref(PidRef :: maybe_pid_ref()) -> maybe_pid_ref().
+destroy_pid_ref(_PidRef) ->
+    erase(?PID_REF).
+
+%%
+%% Context Lifecycle API
+%%
+
+-spec create_worker_context(From, MFA, Nonce) -> pid_ref() | false when
+    From :: pid_ref(), MFA :: mfa(), Nonce :: nonce().
+create_worker_context(From, {M, F, _A}, Nonce) ->
+    case is_enabled() of
+        true ->
+            Type = #rpc_worker{from = From, mod = M, func = F},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_coordinator_context(Httpd, Path) -> pid_ref() | false when
+    Httpd :: #httpd{}, Path :: list().
+create_coordinator_context(#httpd{method = Verb, nonce = Nonce}, Path0) ->
+    case is_enabled() of
+        true ->
+            Path = list_to_binary([$/ | Path0]),
+            Type = #coordinator{method = Verb, path = Path},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_context(Type :: rctx_type(), Nonce :: term()) -> pid_ref() | false.
+create_context(Type, Nonce) ->
+    Rctx = couch_srt_server:new_context(Type, Nonce),
+    PidRef = get_pid_ref(Rctx),
+    set_pid_ref(PidRef),
+    try
+        couch_srt_util:put_delta_a(Rctx),
+        couch_srt_util:put_updated_at(Rctx),
+        couch_srt_server:create_resource(Rctx),
+        couch_srt_logger:track(Rctx),
+        PidRef
+    catch
+        _:_ ->
+            couch_srt_server:destroy_resource(PidRef),
+            %% calling destroy_context(PidRef) clears the tracker too
+            destroy_context(PidRef),
+            false
+    end.
+
+-spec set_context_dbname(DbName :: binary()) -> boolean().
+set_context_dbname(DbName) ->
+    set_context_dbname(DbName, get_pid_ref()).
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: binary(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_dbname(DbName, PidRef).
+
+-spec set_context_handler_fun(Handler) -> boolean() when
+    Handler :: function() | {atom(), atom()}.
+set_context_handler_fun(Handler) ->
+    set_context_handler_fun(Handler, get_pid_ref()).
+
+-spec set_context_handler_fun(Handler, PidRef) -> boolean() when
+    Handler :: function() | {atom(), atom()}, PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun(Fun, PidRef) when is_function(Fun) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            FProps = erlang:fun_info(Fun),
+            Mod = proplists:get_value(module, FProps),
+            Func = proplists:get_value(name, FProps),
+            set_context_handler_fun({Mod, Func}, PidRef)
+    end;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            couch_srt_server:set_context_handler_fun({Mod, Func}, PidRef)
+    end.
+
+%% @equiv set_context_username(User, get_pid_ref())
+set_context_username(User) ->
+    set_context_username(User, get_pid_ref()).
+
+-spec set_context_username(User, PidRef) -> boolean() when
+    User :: null | undefined | #httpd{} | #user_ctx{} | binary(),
+    PidRef :: maybe_pid_ref().
+set_context_username(null, _) ->
+    false;
+set_context_username(_, undefined) ->
+    false;
+set_context_username(#httpd{user_ctx = Ctx}, PidRef) ->
+    set_context_username(Ctx, PidRef);
+set_context_username(#user_ctx{name = Name}, PidRef) ->
+    set_context_username(Name, PidRef);
+set_context_username(UserName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_username(UserName, PidRef).
+
+-spec destroy_context() -> ok.
+destroy_context() ->
+    destroy_context(get_pid_ref()).
+
+-spec destroy_context(PidRef :: maybe_pid_ref()) -> ok.
+destroy_context(undefined) ->
+    ok;
+destroy_context(PidRef) ->
+    %% Stopping the tracker clears the ets entry for PidRef on its way out
+    couch_srt_logger:stop_tracker(),
+    destroy_pid_ref(PidRef),
+    clear_pdict_markers(),
+    ok.
+
+-spec clear_pdict_markers() -> ok.
+clear_pdict_markers() ->
+    ok = lists:foreach(
+        fun
+            ({{csrt, _} = K, _V}) ->
+                erlang:erase(K);
+            (_) ->
+                ok
+        end,
+        erlang:get()
+    ).
+
+%%
+%% Public API
+%%
+
+-spec maybe_track_rexi_init_p({M, F, A}) -> couch_stats:response() when
+    M :: atom(), F :: atom(), A :: non_neg_integer().
+maybe_track_rexi_init_p({M, F, _A}) ->
+    Metric = [M, F, spawned],
+    case couch_srt:should_track_init_p(Metric) of
+        true -> couch_stats:increment_counter(Metric);
+        false -> ok
+    end.
+
+%% Only potentially track positive increments to counters
+-spec maybe_track_local_counter(any(), any()) -> ok.
+maybe_track_local_counter(Name, Val) when is_integer(Val) andalso Val > 0 ->
+    couch_srt:maybe_inc(Name, Val),
+    ok;
+maybe_track_local_counter(_, _) ->
+    ok.
+
+%% @equiv couch_srt_util:is_enabled().
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    couch_srt_util:is_enabled().
+
+%% @equiv couch_srt_util:is_enabled_reporting().
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    couch_srt_util:is_enabled_reporting().
+
+%% @equiv couch_srt_util:is_enabled_rpc_reporting().
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    couch_srt_util:is_enabled_rpc_reporting().
+
+%% @equiv couch_srt_util:is_enabled_init_p().
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    couch_srt_util:is_enabled_init_p().
+
+-spec get_resource() -> maybe_rctx().
+get_resource() ->
+    get_resource(get_pid_ref()).
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(PidRef) ->
+    couch_srt_server:get_resource(PidRef).
+
+%% Log a CSRT report if any filters match
+-spec maybe_report(ReportName :: string(), PidRef :: pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    couch_srt_logger:maybe_report(ReportName, PidRef).
+
+%% Direct report logic skipping should log filters
+-spec do_report(ReportName :: string(), PidRef :: pid_ref()) -> boolean().
+do_report(ReportName, PidRef) ->
+    couch_srt_logger:do_report(ReportName, get_resource(PidRef)).
+
+-spec to_json(Rctx :: maybe_rctx()) -> map() | null.
+to_json(undefined) ->
+    null;
+to_json(Rctx) ->
+    couch_srt_entry:to_json(Rctx).
+
+%%
+%% Stat Collection API
+%%
+
+-spec inc(Key :: rctx_field()) -> non_neg_integer().
+inc(Key) ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key);
+        false ->
+            0
+    end.
+
+-spec inc(Key :: rctx_field(), N :: non_neg_integer()) -> non_neg_integer().
+inc(Key, N) when is_integer(N) andalso N >= 0 ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key, N);
+        false ->
+            0
+    end.
+
+-spec maybe_inc(Stat :: atom(), Val :: non_neg_integer()) -> non_neg_integer().
+maybe_inc(Stat, Val) ->
+    case maps:is_key(Stat, ?STATS_TO_KEYS) of
+        true ->
+            inc(maps:get(Stat, ?STATS_TO_KEYS), Val);
+        false ->
+            0
+    end.
+
+-spec should_track_init_p(Stat :: [atom()]) -> boolean().
+%% "Example to extend CSRT"
+%% should_track_init_p([fabric_rpc, foo, spawned]) ->
+%%    is_enabled_init_p();
+should_track_init_p([fabric_rpc, all_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, changes, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, get_all_security, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, map_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_doc, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_shard, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, reduce_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, update_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p(_Metric) ->
+    false.
+
+-spec ioq_called() -> non_neg_integer().
+ioq_called() ->
+    inc(ioq_calls).
+
+%% we cannot yet use stats couchdb.query_server.*.ddoc_filter because those
+%% are collected in a dedicated process.
+%% TODO: funnel back stats from background worker processes to the RPC worker
+js_filtered(N) ->
+    inc(js_filter),
+    inc(js_filtered_docs, N).
+
+docs_written(N) ->
+    inc(docs_written, N).
+
+-spec accumulate_delta(Delta :: map() | undefined) -> ok.
+accumulate_delta(Delta) when is_map(Delta) ->
+    is_enabled() andalso couch_srt_server:update_counters(get_pid_ref(), Delta),
+    ok;
+accumulate_delta(undefined) ->
+    ok.
+
+-spec make_delta() -> maybe_delta().
+make_delta() ->
+    case is_enabled() of
+        false ->
+            undefined;
+        true ->
+            couch_srt_util:make_delta(get_pid_ref())
+    end.
+
+-spec rctx_delta(TA :: maybe_rctx(), TB :: maybe_rctx()) -> maybe_delta().
+rctx_delta(TA, TB) ->
+    couch_srt_util:rctx_delta(TA, TB).
+
+%%
+%% Aggregate Query API
+%%
+
+-spec active() -> [rctx()].
+active() ->
+    couch_srt_query:active().
+
+-spec active(Type :: json) -> [rctx()].
+active(Type) ->
+    couch_srt_query:active(Type).
+
+-spec active_coordinators() -> [coordinator_rctx()].
+active_coordinators() ->
+    couch_srt_query:active_coordinators().
+
+%% TODO: cleanup json logic here
+-spec active_coordinators(Type :: json) -> [coordinator_rctx()].
+active_coordinators(Type) ->
+    couch_srt_query:active_coordinators(Type).
+
+-spec active_workers() -> [rpc_worker_rctx()].
+active_workers() ->
+    couch_srt_query:active_workers().
+
+-spec active_workers(Type :: json) -> [rpc_worker_rctx()].
+active_workers(Type) ->
+    couch_srt_query:active_workers(Type).
+
+find_by_nonce(Nonce) ->
+    couch_srt_query:find_by_nonce(Nonce).
+
+find_by_pid(Pid) ->
+    couch_srt_query:find_by_pid(Pid).
+
+find_by_pidref(PidRef) ->
+    couch_srt_query:find_by_pidref(PidRef).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_query:find_workers_by_pidref(PidRef).
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    couch_srt_logger:pid_ref_matchspec(AttrName).
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    couch_srt_logger:pid_ref_attrs(AttrName).
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with couch_srt_logger:pid_ref_attrs/1, and
+%% returning on pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    couch_srt_logger:proc_window(AttrName, Num, Time).
+
+-spec query_matcher(MatcherName :: matcher_name()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName) ->
+    couch_srt_query:query_matcher(MatcherName).
+
+-spec query_matcher(MatcherName :: matcher_name(), Limit :: pos_integer()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName, Limit) ->
+    couch_srt_query:query_matcher(MatcherName, Limit).
+
+%%
+%% Delta API
+%%
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, Delta) ->
+    couch_srt_util:add_delta(T, Delta).
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta(T) ->
+    couch_srt_util:extract_delta(T).
+
+-spec get_delta() -> tagged_delta().
+get_delta() ->
+    couch_srt_util:get_delta(get_pid_ref()).
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    couch_srt_util:maybe_add_delta(T).
+
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, Delta) ->
+    couch_srt_util:maybe_add_delta(T, Delta).
+
+%%
+%% Query API functions
+%%
+%%
+
+%% @doc Construct query from the expressions.
+%% There are following types of expressions allowed in the query.
+%%   <li>group_by/1 @see group_by/1</li>
+%%   <li>group_by/2 @see group_by/1</li>
+%%   <li>sort_by/1 @see sort_by/1</li>
+%%   <li>count_by/1 @see count_by/1</li>
+%%   <li>options/1 @see options/1</li>
+%%   <li>from/1 @see from/1</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec query(QueryExpression :: [query_expression()]) ->
+    query() | {error, any()}.
+query(QueryExpression) ->
+    couch_srt_query:query(QueryExpression).
+
+%% @doc Specify the matcher to use for the query.
+%% If atom 'all' is used then all entries would be in the scope of the query.
+%% Also the use of 'all' makes the query 'unsafe'. Because it scans through all entries
+%% and can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    from("docs_read")
+%% ]),
+%% </code>
+%% @end
+-spec from(MatcherNameOrAll :: string() | all) ->
+    query_expression() | {error, any()}.
+from(MatcherNameOrAll) ->
+    couch_srt_query:from(MatcherNameOrAll).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+group_by(AggregationKeys) ->
+    couch_srt_query:group_by(AggregationKeys).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+group_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:group_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+sort_by(AggregationKeys) ->
+    couch_srt_query:sort_by(AggregationKeys).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+sort_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:sort_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'count_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    count_by(username)
+%% ]),
+%% </code>
+%% @end
+-spec count_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+count_by(AggregationKeys) ->
+    couch_srt_query:count_by(AggregationKeys).
+
+%% @doc Construct 'options' query expression.
+%% There are following types of expressions allowed in the query.
+%%   <li>unlimited/0 @see unlimited/0 (cannot be used with 'with_limit/1')</li>
+%%   <li>with_limit/1 @see with_limit/1 (cannot be used with 'unlimited/0')</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      ...
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec options([query_option()]) ->
+    query_expression() | {error, any()}.
+options(OptionsExpression) ->
+    couch_srt_query:options(OptionsExpression).
+
+%% @doc Enable unlimited number of results from the query.
+%% The use of 'unlimited' makes the query 'unsafe'. Because it can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      unlimited()
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec unlimited() ->
+    query_expression().
+unlimited() ->
+    couch_srt_query:unlimited().
+
+%% @doc Set limit on number of results returned from the query.
+%% The construction of the query fail if the 'limit' is greater than
+%% allowed for this cluster.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      with_limit(100)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec with_limit(Limit :: pos_integer()) ->
+    query_expression() | {error, any()}.
+with_limit(Limit) ->
+    couch_srt_query:with_limit(Limit).
+
+%% @doc Executes provided query. Only 'safe' queries can be executed using 'run'.
+%% The query considered 'unsafe' if any of the conditions bellow are met:
+%%   <li>Query uses 'unlimited/0'</li>
+%%   <li>Query uses 'from(all)'</li>
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% run(Q)
+%% </code>
+%% @end
+-spec run(query()) ->
+    {ok, [{aggregation_key(), pos_integer()}]}
+    | {limit, [{aggregation_key(), pos_integer()}]}.
+run(Query) ->

Review Comment:
   At first sight this seems like overkill for an off-to-the-side stats aggregator API meant to be called by server admin operator to figure out their resource usage. I can see this being needed for a public-facing high rate API like changes or all_docs but here overcomplicates things. Maybe just let the operator pass in a `limit` parameter and have single handler function?



##########
src/chttpd/src/chttpd_httpd_handlers.erl:
##########
@@ -20,6 +20,7 @@ url_handler(<<"_utils">>) -> fun chttpd_misc:handle_utils_dir_req/1;
 url_handler(<<"_all_dbs">>) -> fun chttpd_misc:handle_all_dbs_req/1;
 url_handler(<<"_dbs_info">>) -> fun chttpd_misc:handle_dbs_info_req/1;
 url_handler(<<"_active_tasks">>) -> fun chttpd_misc:handle_task_status_req/1;
+url_handler(<<"_active_resources">>) -> fun couch_srt_httpd:handle_resource_status_req/1;

Review Comment:
   We have an API documentation section, we should probably document what the requests look like there



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.
+
+### #rctx.rows_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([fabric_rpc, changes, processed])
+> also Tracking `couch_stats:increment_counter([fabric_rpc, view, rows_read])
+
+A value tracking multiple possible metrics corresponding to rows streamed in
+aggregate operations. This is used for view_rows/changes_rows/all_docs/etc.
+
+### #rctx.changes_returned = 0 :: non_neg_integer() | '_',
+
+The number of `fabric_rpc:changes_row/2` invocations induced by this context,
+specifically tracking the number of changes rows streamed back to the client
+requeest, allowing for distinguishing between the number of changes processed to
+fulfill a request versus the number actually returned in the http response.
+
+### #rctx.ioq_calls = 0 :: non_neg_integer() | '_',
+
+A phony metric counting invocations of `ioq:call/3` induced by this context. As
+with `#rctx.docs_written`, we need a proxy metric to reperesent these calls
+until CSRT context passing is supported so that the `ioq_server` pid and return
+its own delta back to the worker pid.
+
+### #rctx.js_filter = 0 :: non_neg_integer() | '_',
+
+A phony metric counting the number of `couch_query_servers:filter_docs_int/5`
+(eg ddoc_prompt) invocations induced by this context. This is called by way of
+`couch_srt:js_filtered(length(JsonDocs))` which both increments `js_filter` by 1, and
+`js_filtered_docs` by the length of the docs so we can track magnitude of docs
+and doc revs being filtered.
+
+### #rctx.js_filtered_docs = 0 :: non_neg_integer() | '_',
+
+A phony metric counting the quantity of documents filtered by way of
+`couch_query_servers:filter_docs_int/5` (eg ddoc_prompt) invocations induced by
+this context. This is called by way of `couch_srt:js_filtered(length(JsonDocs))`
+which both increments `#rctx.js_filter` by 1, and `#rctx.js_filtered_docs` by
+the length of the docs so we can track magnitude of docs and doc revs being
+filtered.

Review Comment:
   Wonder why does this one need to be phony? Is it because it's the os proc gen_server bumping those metrics? Maybe those stats bump should be pulled out to happen in the caller and then we don't have to have phony metrics



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.

Review Comment:
   Nice! This is good stuff, a small example is worth a 1000 words!



##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case couch_srt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = couch_srt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), couch_srt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(Msg, From, St) ->
+    {stop, {unknown_call, Msg, From}, St}.
+
+handle_cast(Msg, St) ->
+    {stop, {unknown_cast, Msg}, St}.
+
+handle_info(restart_config_listener, St) ->
+    ok = subscribe_changes(),
+    {noreply, St};
+handle_info(Msg, St) ->
+    {stop, {unknown_info, Msg}, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbnames_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbnames_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% so we must convert the provided Threshold from milliseconds to native
+    %% representation which then allows us to match against entries where the
+    %% time delta between the started_at and updated_at is greater than the
+    %% native converted threshold.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using couch_srt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = couch_srt_util:tnow(), timer:sleep(10000), T1 = couch_srt_util:tnow(),
+    %%     erlang:convert_time_unit(T1 - T0, native, millisecond).
+    %% 10000
+
+    Unit = millisecond,

Review Comment:
   Nit: technically both `millisecond` and `native` are "units". Could just put the `millisecond` inline in the `convert_time_unit(...)` call and save an extra line of code.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.

Review Comment:
   Nit: extra space after `counting`?



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the

Review Comment:
   Minor nit: use backquotes for code references `started_at`, `updated_at`



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.
+
+### #rctx.rows_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([fabric_rpc, changes, processed])
+> also Tracking `couch_stats:increment_counter([fabric_rpc, view, rows_read])

Review Comment:
   The quote / snippet looks a bit odd with just one backquote 
   
   <img width="544" height="43" alt="Screenshot 2025-07-23 at 5 48 02 PM" src="https://github.com/user-attachments/assets/70b185c7-c70c-4d1f-96c9-ceca5615862f" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226712881


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a

Review Comment:
   Users will likely not read the macros from the code so better to use plain string names. Config settings should also probably also go into the main docs area for config values. There is some special syntax there to cross link them and such and some default formatting applied and such.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a

Review Comment:
   Users will likely not read the macros from the code so better to use plain string names. Config settings should also probably also go into the main docs area for config values. There is some special syntax there to cross link them and such and some default formatting applied.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236536174


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   I see now that is confusing. Here is how I came about this spec. The query_result function return JSON as expected by `jiffy:encode/1` (`jiffy:json_value()`). However `jiffy:json_value()` is opaque and cannot serve as a documentation. So my first idea was to have a comment which shows the structure of the result. However maintaining the comment is harder because compiler doesn't check the comments. So I invented `-type json_spec(_Spec) :: term().` This type *ignores* what's inside. It is meant for documenting JSON structure. I thought it would be great to use across codebase to provide example of the data structures used by functions. So I used the types which make sense to document JSON. I.e. `string()` in this context refer to the JSON string. I could change it to `jiffy:json_string().` although I need to check whether compiler would like it or not (the problem might be that `jiffy` doesn't export its types).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236907199


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a

Review Comment:
   Yeah agreed, I've removed all the macros and ported most of the docs into the formal `src/docs` Sphinx app, with remaining docs in CSRT.md being more focused on internals and code structure as well as next steps and other documentation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2323547328


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.

Review Comment:
   We can still catch `ets:badarg` and I am not sure if ets:insert would result in a `system_limit` that's for incrementing a counter or reaching nesting limit with a matchspec.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2257306060


##########
src/couch_srt/README.md:
##########
@@ -206,20 +207,20 @@ The query and HTTP API's are well documented and tested (h/t @iilyak) and
 provide an excellent overview of the interaction patterns and query capabilities
 of CSRT. Those can be found at:
 
-* `csrt_query.erl` "Query API functions"
-  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/csrt_query.erl#L319-L674
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674

Review Comment:
   I did update all links



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2337329016


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of

Review Comment:
   Addressed in 36feba6778c47e0bd614a2434a83f89b34bcb034. Although I went with 5000. The row tuples are small. I think we should be ok with bigger batches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349709583


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   Thank you Nick. I did switch to `gb_sets` in c5b9fe0e36862e1ead86a322703d7b5e2625d0b2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3300270434

   > @nickva
   >  I also tried it out locally in my dev cluster but couldn't get anything to show up in the logs.
   
   I did repeat the steps you did locally.
   
   ```
   (node1@127.0.0.1)2> config:set("csrt", "enable", "true").
   ok
   
   (node1@127.0.0.1)3> config:set("csrt", "enable_init_p", "true").
   ok
   
   (node1@127.0.0.1)4> config:set("csrt", "enable_reporting", "true").
   ok
   
   (node1@127.0.0.1)5> config:set("csrt_logger.matchers_enabled", "all_coordinators", "true").
   
   ok
   (node1@127.0.0.1)6> config:set("csrt_logger.matchers_enabled", "docs_read", "true").
   ok
   
   (node1@127.0.0.1)7> config:set("csrt_logger.matchers_threshold", "docs_read", "50").
   ok
   
   (node1@127.0.0.1)8> config:set("csrt_logger.matchers_enabled", "changes_processed", "true").
   ok
   
   (node1@127.0.0.1)9> config:set("csrt_logger.matchers_threshold", "changes_processed", "23").
   ```
   
   Except I didn't create new database. I queried `_replicator` db.
   
   ```
   while true; do curl -u adm:pass http://127.0.0.1:15984/_replicator/_changes > /dev/null ; sleep 5 ; done
   ```
   
   This is what I see in the logs
   
   ```
   [notice] 2025-09-16T20:30:11.007168Z node1@127.0.0.1 <0.10531.0> ef994f45d5 127.0.0.1:15984 127.0.0.1 adm GET /_replicator/_changes 200 ok 1
   [report] 2025-09-16T20:30:11.007228Z node1@127.0.0.1 <0.10580.0> -------- [csrt-pid-usage-lifetime db_open=5 dbname="_replicator" nonce="ef994f45d5" pid_ref="<0.10531.0>:#Ref<0.3096609386.3958636548.180199>" started_at="2025-09-16T20:30:11.006z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/_replicator/_changes" updated_at="2025-09-16T20:30:11.006z" username="adm"]
   [notice] 2025-09-16T20:30:16.056722Z node1@127.0.0.1 <0.10689.0> 9c9fcd161b 127.0.0.1:15984 127.0.0.1 adm GET /_replicator/_changes 200 ok 2
   [report] 2025-09-16T20:30:16.056874Z node1@127.0.0.1 <0.10738.0> -------- [csrt-pid-usage-lifetime db_open=5 dbname="_replicator" nonce="9c9fcd161b" pid_ref="<0.10689.0>:#Ref<0.3096609386.3958636546.179197>" started_at="2025-09-16T20:30:16.054z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/_replicator/_changes" updated_at="2025-09-16T20:30:16.056z" username="adm"]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221180861


##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->
+            R
+        end
+    ).
+
+-spec matcher_on_ioq_calls(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_ioq_calls(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{ioq_calls = IOQCalls} = R) when IOQCalls >= Threshold -> R end).
+
+%%-spec matcher_for_rctx_field(Field :: rctx_field()) -> ets:match_spec().
+%%matcher_for_rctx_field() ->
+%%    #{size := Size0, fields := Fields} = csrt_entry:record_info(),
+%%    %% Subtract 1 as record_info size includes tuple record name
+%%    %% erlang:list_to_tuple([rctx | lists:duplicate(maps:get(size, csrt_entry:record_info()), '_')])
+%%    Size = Size - 1,
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> matcher() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    #{field_idx := FieldIdx} = csrt_entry:record_info(),
+    RctxMatch0 = #rctx{_ = '_'},
+    RctxMatch1 = setelement(maps:get(pid_ref, FieldIdx) + 1, RctxMatch0, '$1'),
+    RctxMatch = setelement(maps:get(AttrName, FieldIdx) + 1, RctxMatch1, '$2'),
+    MatchSpec = [{RctxMatch, [], [{{'$1', '$2'}}]}],
+    {MatchSpec, ets:match_spec_compile(MatchSpec)}.
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> list() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    {MatchSpec, _CompMatch} = pid_ref_matchspec(AttrName),
+    %% Base fields at least an empty list, but we could add more info here.
+    %% The recon typespec is an improper list of the form:
+    %%Base = [Name | [{current_function, mfa()} | {initial_call, mfa()}]],
+    Base = [],
+    [{PidRef, Val, Base} || {PidRef, Val} <- ets:select(?CSRT_ETS, MatchSpec)].
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+
+-spec add_matcher(Name, MSpec, Matchers) ->
+    {ok, matchers()} | {error, {invalid_ms, string(), ets:match_spec()}}
+when
+    Name :: string(), MSpec :: ets:match_spec(), Matchers :: matchers().
+add_matcher(Name, MSpec, Matchers) ->
+    try ets:match_spec_compile(MSpec) of
+        CompMSpec ->
+            %% TODO: handle already registered name case
+            Matchers1 = maps:put(Name, {MSpec, CompMSpec}, Matchers),
+            {ok, Matchers1}
+    catch
+        error:badarg ->
+            {error, {invalid_ms, Name, MSpec}}
+    end.
+
+-spec set_matchers_term(Matchers :: matchers()) -> ok.
+set_matchers_term(Matchers) when is_map(Matchers) ->
+    persistent_term:put(?MATCHERS_KEY, Matchers).
+
+-spec initialize_matchers(RegisteredMatchers :: map()) -> ok.
+initialize_matchers(RegisteredMatchers) when is_map(RegisteredMatchers) ->
+    %% Standard matchers to conditionally enable
+    DefaultMatchers = [
+        {docs_read, fun matcher_on_docs_read/1, 1000},
+        {rows_read, fun matcher_on_rows_read/1, 1000},
+        {docs_written, fun matcher_on_docs_written/1, 500},
+        %% long_reqs Threshold in milliseconds
+        {long_reqs, fun matcher_on_long_reqs/1, 60000},
+        {changes_processed, fun matcher_on_changes_processed/1, 1000},
+        {ioq_calls, fun matcher_on_ioq_calls/1, 10000}
+    ],
+
+    %% Add enabled Matchers for standard matchers
+    Matchers = lists:foldl(
+        fun({Name0, MatchGenFunc, Threshold0}, Matchers0) when is_atom(Name0) ->
+            Name = atom_to_list(Name0),
+            case matcher_enabled(Name) of
+                true ->
+                    %% Wrap in a try-catch to handle MatcherGen errors
+                    try
+                        Threshold = matcher_threshold(Name, Threshold0),
+                        case add_matcher(Name, MatchGenFunc(Threshold), Matchers0) of
+                            {ok, Matchers1} ->
+                                Matchers1;
+                            {error, {invalid_ms, NameE, MSpecE}} ->
+                                couch_log:warning("[~p] Failed to initialize matcher[~p]: ~p", [
+                                    ?MODULE, NameE, MSpecE
+                                ]),
+                                Matchers0
+                        end
+                    catch
+                        _:_ ->
+                            Matchers0
+                    end;
+                false ->
+                    couch_log:warning("[~p] Failed to initialize matcher: ~p", [
+                        ?MODULE, Name
+                    ]),
+                    Matchers0
+            end
+        end,
+        #{},
+        DefaultMatchers
+    ),
+
+    %% Add additional dbname_io matchers
+    Matchers1 = lists:foldl(
+        fun({Dbname, Value}, Matchers0) ->
+            try list_to_integer(Value) of
+                Threshold when Threshold > 0 ->
+                    Name = "dbname_io__" ++ Dbname ++ "__" ++ Value,
+                    DbnameB = list_to_binary(Dbname),
+                    MSpec = matcher_on_dbname_io_threshold(DbnameB, Threshold),
+                    case add_matcher(Name, MSpec, Matchers0) of
+                        {ok, Matchers1} ->
+                            Matchers1;
+                        {error, {invalid_ms, NameE, MSpecE}} ->
+                            couch_log:warning("[~p] Failed to initialize matcher[~p]: ~p", [
+                                ?MODULE, NameE, MSpecE
+                            ]),
+                            Matchers0
+                    end;
+                _ ->
+                    Matchers0
+            catch
+                error:badarg ->
+                    couch_log:warning("[~p] Failed to initialize dbname io matcher on[~p]: ~p", [
+                        ?MODULE, Dbname, Value
+                    ]),
+                    Matchers0
+            end
+        end,
+        Matchers,
+        config:get(?CSRT_MATCHERS_DBNAMES)
+    ),
+
+    %% Finally, merge in the dynamically registered matchers, with priority
+    Matchers2 = maps:merge(Matchers1, RegisteredMatchers),
+
+    couch_log:notice("Initialized ~p CSRT Logger matchers", [maps:size(Matchers2)]),
+    set_matchers_term(Matchers2),
+    ok.
+
+-spec matcher_enabled(Name :: string()) -> boolean().
+matcher_enabled(Name) when is_list(Name) ->
+    %% TODO: fix
+    %% config:get_boolean(?CSRT_MATCHERS_ENABLED, Name, false).

Review Comment:
   No, this is the default toggle enablement to swap prior to merging.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221251431


##########
src/couch_stats/src/couch_stats.erl:
##########
@@ -49,6 +54,10 @@ increment_counter(Name) ->
 
 -spec increment_counter(any(), pos_integer()) -> response().
 increment_counter(Name, Value) ->
+    %% Should maybe_track_local happen before or after notify?
+    %% If after, only currently tracked metrics declared in the app's
+    %% stats_description.cfg will be trackable locally. Pros/cons.
+    ok = maybe_track_local_counter(Name, Value),

Review Comment:
   I've addressed this more in my top level PR comment in https://github.com/apache/couchdb/pull/5602#issuecomment-3100600679



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223241247


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```

Review Comment:
   I think anyone who would understand what the details are would likely just look at the code directly. A user who wants to setup tracing or resource monitoring is likely to get more confused by the type detailsl.



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```

Review Comment:
   I think anyone who would understand what the details are would likely just look at the code directly. A user who wants to setup tracing or resource monitoring is likely to get more confused by the type details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236412869


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").

Review Comment:
   Yeah, I had duplicating strings, so I did that as a way to define them at the top of the file so folks could perhaps more easily see the different groups, but porting these settings over to the config docs eliminated that issue while also adding proper documentation hahah. ALl the macros should be out of the rst and md docs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2217007054


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([

Review Comment:
   I think this list is outdated.
   
   ```erlang
   % Aggregate Query API
   -export([
       active/0,
       active/1,
       active_coordinators/0,
       active_coordinators/1,
       active_workers/0,
       active_workers/1,
       find_by_nonce/1,
       find_by_pid/1,
       find_by_pidref/1,
       find_workers_by_pidref/1,
       query_matcher/1,
       query_matcher/2
   ]).
   
   -export([
       query/1,
       from/1,
       group_by/1,
       group_by/2,
       sort_by/1,
       sort_by/2,
       count_by/1,
       options/1,
       unlimited/0,
       with_limit/1,
   
       run/1,
       unsafe_run/1
   ]).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2323445643


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->

Review Comment:
   > this should probably be error:badarg only, what other exceptions would we catch here?
   
   `system_limit`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316466921


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need

Review Comment:
   Thanks for adding the docs, they look nice!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317581417


##########
src/docs/src/config/csrt.rst:
##########
@@ -0,0 +1,587 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. default-domain:: config
+.. highlight:: ini
+
+.. _config-csrt:
+
+==========================================
+Couch Stats Resource Tracker (CSRT) Config
+==========================================
+
+CSRT configuration options and overview.
+
+.. seealso::
+
+    :doc:`/csrt/index`
+
+CSRT config
+===========
+
+This section contains the top level enablement and configuration options for CSRT.
+
+.. config:section:: csrt :: CSRT Primary Configuration
+
+    .. config:option:: enable :: Enable CSRT data collection and RPC deltas
+
+        Core enablement toggle for CSRT, defaults to false. Enabling this
+        setting initiates local CSRT stats collection as well as shipping deltas
+        in RPC responses to accumulate in the coordinator.
+
+        This does *not* trigger the new RPC spawn metrics, and it does not
+        enable reporting for any of the rctx types.
+
+        .. warning::
+
+            You *MUST* have all nodes in the cluster running a CSRT aware
+            CouchDB *before* you enable it on any node, otherwise the old
+            version nodes won't know how to handle the new RPC formats
+            including an embedded Delta payload.
+
+        Top level CSRT enablement for local data collection and RPC deltas::
+
+            [csrt]
+            enable = false
+
+    .. config:option:: enable_init_p :: Enable RPC spawn metric tracking
+
+        Enablement of tracking new metric counters for different `fabric_rpc` operations
+        spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+        inducing database RPC operations within CouchDB, and these init_p metrics aim to
+        provide node level understandings of the workloads being induced by other
+        coordinator processes. This is especially relevant for databases on subsets of a
+        cluster resulting in non-uniform workloads, these metrics are tailored to
+        provide insight into what work is being spawned on each node in the cluster as a
+        function of time.
+
+        Enablement for tracking counts of spawned RPC workers::
+
+            [csrt]
+            enable_init_p = false
+
+    .. config:option:: enable_reporting :: Enable CSRT Process Lifecyle Reports
+
+        This is the primary toggle for enabling CSRT process lifetime reports
+        containing detailed information about the quantity of work induced by
+        the given request/worker/etc. This is the top level toggle for enabling
+        _any_ reporting, and there also exists
+        :config:option:`csrt/enable_rpc_reporting` to disable the reporting of
+        any individual RPC workers, leaving the coordinator responsible of
+        generating a report with the accumulated deltas.
+
+        .. note::
+
+            Note that this setting toggles whether or not to generate process
+            lifecycle reports, but no reports will be generated until logger
+            matchers have been enabled that trigger a match on CSRT contexts
+            that have surpassed the configured thresholds.
+
+        Top level toggle for whether any process lifecycle reports are generated::
+
+            [csrt]
+            enable_reporting = false
+
+    .. config:option:: enable_rpc_reporting :: Enable RPC process lifecyle reports
+
+        This enables the possibility of RPC workers generating reports. They
+        still need to hit the configured thresholds to induce a report, but
+        this will generate CSRT process lifetime reports for individual RPC
+        workers that trigger the configured logger thresholds. This allows for
+        quantifying per node resource usage when desired, as otherwise the
+        reports are at the http request level and don't provide per node stats.
+
+        The key idea here is that having RPC level CSRT process lifetime
+        reporting is incredibly useful, but can also generate large quantities
+        of data. For example, a view query on a Q=64 database will stream
+        results from 64 shard replicas, resulting in at least 64 RPC reports,
+        plus any that might have been generated from RPC workers that "lost"
+        the race for shard replica. This is very useful, but a lot of data
+        given the verbose nature of funneling it through the RSyslog reports,
+        however, the ability to write directly to something like ClickHouse or
+        another columnar store would be great.
+
+        Until there's an efficient storage mechanism to stream the results to,
+        the rsyslog entries work great and are very practical, but care must be
+        taken to not generate too much data for aggregate queries as they
+        generate at least `Qx` more report than an individual report per http
+        request from the coordinator.  This setting exists as a way to either
+        a) utilize the logger matcher configured thresholds to allow for _any_
+        rctx's to be recorded when they induce heavy operations, either
+        Coordinator or RPC worker; or b) to _only_ log workloads at the
+        coordinator level.
+
+        .. note::
+
+            This setting exists because we lack an expressive enough config
+            declaration to easily chain the matchspec constructions as
+            `ets:fun2ms/1` is a special compile time parse transform macro that
+            requires the full definition to be specified directly, it cannot
+            be interactively constructed. That said, you _can_ register matchers
+            through `remsh` with more specific and fine grained pattern matching,
+            and a more expressive system for defining matchers are being
+            explored.
+
+        .. warning::
+
+            Enabling this setting *will* generate considerably more logs! Specifically, for aggregate queries and database operations, this will generate `Q` * `N` times more logs than a singular doc request taking only `N` inreacting with a singular shard range. See the note above about this being a temporary setting during the experimental stages of CSRT.
+
+        Toggle to enable possibility of RPC process lifecycle reports::
+
+            [csrt]
+            enable_rpc_reporting = false
+
+    .. config:option:: should_truncate_reports :: truncate zero values from lifecyle reports
+
+        enables truncation of the csrt process lifetime reports to not include
+        any fields that are zero at the end of process lifetime, eg don't
+        include `js_filter=0` in the report if the request did not induce
+        javascript filtering.
+
+        this can be disabled if you really care about consistent fields in the
+        report logs, but this is a log space saving mechanism, similar to
+        disabling rpc reporting by default, as its a simple way to reduce
+        overall volume
+
+        Truncate zero values from process lifecycle reports, enabled by default:
+
+            [csrt]
+            should_truncate_reports = true
+
+    .. config:option:: query_limit :: Maximum quantity of rows to return in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_limit = 100
+
+    .. config:option:: query_cardinality_limit :: Maximum quantity of rows to allow in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_cardinality_limit = 10000
+
+.. _csrt-logger-matcher-configuration:
+
+CSRT Logger Matcher Configuration
+=================================
+
+There are currently eight builtin default logger matchers designed to make it
+easy to do filtering on heavy resource usage inducing and long running
+requests. These are designed as a simple baseline of useful matchers, declared
+in a manner amenable to `default.ini` based constructs. More expressive matcher
+declarations are being explored, and matchers of arbitrary complexity can be
+registered directly through `remsh`. The default matchers are all designed around
+an integer config Threshold that triggers on a specific field, eg docs read, or

Review Comment:
   Probably don't need to capitalize `T` in `threshold`. Or if it's a variable in the code use double backticks for a monspace font.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317659453


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->

Review Comment:
   use `error:badarg` like mentioned above



##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->

Review Comment:
   this should probably be `error:badarg` only, what other exceptions would we catch here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3255347111

   Thanks for taking another look, @nickva!
   
   That's odd the reports aren't showing for you, the config settings you listed look okay, and `couch_srt_logger` has a config:listener to reload the matchers on changes to any of the logger config groups https://github.com/apache/couchdb/blob/couch-stats-resource-tracker-v3-rebase-main/src/couch_srt/src/couch_srt_logger.erl#L588-L598 so that's odd you're not seeing the changes picked up.
   
   The first four commands you did _should_ bootstrap the system into generating reports, eg these settings:
   
   ```
   (node1@127.0.0.1)2> config:set("csrt", "enable", "true").
   ok
   
   (node1@127.0.0.1)3> config:set("csrt", "enable_init_p", "true").
   ok
   
   (node1@127.0.0.1)4> config:set("csrt", "enable_reporting", "true").
   ok
   
   (node1@127.0.0.1)5> config:set("csrt_logger.matchers_enabled", "all_coordinators", "true").
   
   ok
   ```
   
   I just did a fresh clone and with the following diff I'm properly getting reports generated like:
   
   > [notice] 2025-09-04T19:43:09.288686Z node1@127.0.0.1 <0.11224.0> 381d115cad localhost:15984 127.0.0.1 adm GET /_all_dbs 200 ok 3
   > [report] 2025-09-04T19:43:09.288790Z node1@127.0.0.1 <0.11274.0> -------- [csrt-pid-usage-lifetime db_open=5 get_kv_node=2 nonce="381d115cad" pid_ref="<0.11224.0>:#Ref<0.3500684338.1004273667.225747>" rows_read=2 started_at="2025-09-04T19:43:09.292z" type="coordinator-{chttpd_misc:handle_all_dbs_req}:GET:/_all_dbs" updated_at="2025-09-04T19:43:09.295z" username="adm"]
   
   ```diff
   (chewbranca)-(jobs:0)-(/tmp/couchdb)
   (! 15428)-> git diff
   diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
   index b2ffa87b7..3cfe9d1aa 100644
   --- a/rel/overlay/etc/default.ini
   +++ b/rel/overlay/etc/default.ini
   @@ -1153,9 +1153,9 @@ url = {{nouveau_url}}
    
    ; Couch Stats Resource Tracker (CSRT)
    [csrt]
   -;enable = false
   -;enable_init_p = false
   -;enable_reporting = false
   +enable = true
   +enable_init_p = true
   +enable_reporting = true
    ;enable_rpc_reporting = false
    
    ; Truncate reports to not include zero values for counter fields. This is a
   @@ -1223,7 +1223,7 @@ url = {{nouveau_url}}
    ; CSRT default matchers - enablement configuration
    ; The default CSRT loggers can be individually enabled below
    [csrt_logger.matchers_enabled]
   -;all_coordinators = false
   +all_coordinators = true
    ;all_rpc_workers = false
    ;docs_read = false
    ;rows_read = false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317648749


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of

Review Comment:
   How large of difference does each method have compared to another? Instead of keeping both around wonder if there is a middle-ground of querying in smaller batches of limit = 500 and doing a bunch of them in a row until we reach the users's `limit`. It just seems like a lot of complexity added for a method that might be called once in a while by a single operator debugging or reporting a large io usage or investigating some performance issues?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2257151749


##########
src/chttpd/src/chttpd_httpd_handlers.erl:
##########
@@ -20,6 +20,7 @@ url_handler(<<"_utils">>) -> fun chttpd_misc:handle_utils_dir_req/1;
 url_handler(<<"_all_dbs">>) -> fun chttpd_misc:handle_all_dbs_req/1;
 url_handler(<<"_dbs_info">>) -> fun chttpd_misc:handle_dbs_info_req/1;
 url_handler(<<"_active_tasks">>) -> fun chttpd_misc:handle_task_status_req/1;
+url_handler(<<"_active_resources">>) -> fun couch_srt_httpd:handle_resource_status_req/1;

Review Comment:
   > I'm hoping @iilyak can update those docs with his HTTP API updates.
   
   Docs are added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349759708


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   I would even skip the whole update_topK function and the record and just use the topk from above as is. There is no need to keeping track of size as gb data structure does that already, it's fine adding and removing an item in the same update, it's a log k  kind of a deal and log k should already be a small constant anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220242566


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   They are not arbitrary. The imported ones are part of query DSL. So we can say things like:
   
   ```erlang
               query([
                   from("docs_read"),
                   group_by(<<"username">>, <<"ioq_calls">>),
                   options([
                       unlimited()
                   ])
               ])
   ```
   
   Otherwise the query wouldn't be that readable.
   
   ```erlang
               csrt:query([
                   csrt:from("docs_read"),
                   csrt:group_by(<<"username">>, <<"ioq_calls">>),
                   csrt:options([
                       csrt:unlimited()
                   ])
               ])
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223679089


##########
src/rexi/src/rexi_monitor.erl:
##########
@@ -35,6 +35,7 @@ start(Procs) ->
 %% messages from our mailbox.
 -spec stop(pid()) -> ok.
 stop(MonitoringPid) ->
+    unlink(MonitoringPid),

Review Comment:
   This was an artifact of earlier implementation in https://github.com/apache/couchdb/pull/4812 that dropped the use of selective receives and needed to handle all of the messages, which exposed a number of places where we get unexpected messages in the mailbox. I found the old issues discussing this bug and others, many of which since have been fixed, and I believe this one included. I'll drop this out of the PR. Here's the old issues:
   
   * https://github.com/apache/couchdb/issues/5127
   * https://github.com/apache/couchdb/issues/5122
   * https://github.com/apache/couchdb/issues/4909
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220346513


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   That could make sense in general but I think it can still use some cleanup in this case.
   
   * For instance I see `{Keys, csrt:query([from(MatcherName), count_by(Keys)])};` but then also ` {Keys, query([from(MatcherName), sort_by(Keys, CounterKey)])};` so it's mix of both
   
   *  `couch_util:value(...)` doesn't have the shortened imported format but seems just as verbose as `csrt:from(...)`
   
   * `with_limit/1` doesn't seem to be used but is imported
   
   There are 17 lines of imports but only 3 lines of `query(...)` calls so it seems overkill in this module as at least?
   
   Sometimes to avoid imports if we're really repeating the same call too often we can make a few short local functions. Say, if we call `erlang:monotonic_time()` then could make local one-line function called `tnow() -> erlang:monotonic_time()` then we can call that 20 or 30 times and it looks a bit better.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221208922


##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format
+-spec tutc(Time :: integer()) -> calendar:rfc3339_string().
+tutc(Time0) when is_integer(Time0) ->
+    Unit = millisecond,
+    Time1 = Time0 + erlang:time_offset(),
+    Time = erlang:convert_time_unit(Time1, native, Unit),
+    calendar:system_time_to_rfc3339(Time, [{unit, Unit}, {offset, "z"}]).
+
+%% Returns dt (delta time) in microseconds
+%% @equiv make_dt(A, B, microsecond)
+-spec make_dt(A, B) -> pos_integer() when
+    A :: integer(),
+    B :: integer().
+make_dt(A, B) ->
+    make_dt(A, B, microsecond).
+
+%% Returns monotonic dt (delta time) in specified time_unit()
+-spec make_dt(A, B, Unit) -> pos_integer() when
+    A :: integer(),
+    B :: integer(),
+    Unit :: erlang:time_unit().
+make_dt(A, A, _Unit) when is_integer(A) ->
+    %% Handle edge case when monotonic_time()'s are equal
+    %% Always return a non zero value so we don't divide by zero
+    %% This always returns 1, independent of unit, as that's the smallest
+    %% possible positive integer value delta.
+    1;
+make_dt(A, B, Unit) when is_integer(A) andalso is_integer(B) andalso B > A ->
+    case erlang:convert_time_unit(abs(B - A), native, Unit) of

Review Comment:
   Yes, `abs` is necessary, see my other comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221283029


##########
src/rexi/src/rexi_monitor.erl:
##########
@@ -35,6 +35,7 @@ start(Procs) ->
 %% messages from our mailbox.
 -spec stop(pid()) -> ok.
 stop(MonitoringPid) ->
+    unlink(MonitoringPid),

Review Comment:
   Not unlinking is a bug which consumed many many hours of my life.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2219768134


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true

Review Comment:
   In the default ini we normally don't have actual value, just defaults from the code. 



##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true

Review Comment:
   In the default ini we normally don't have actual values, just defaults from the code. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221243951


##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format
+-spec tutc(Time :: integer()) -> calendar:rfc3339_string().
+tutc(Time0) when is_integer(Time0) ->
+    Unit = millisecond,
+    Time1 = Time0 + erlang:time_offset(),
+    Time = erlang:convert_time_unit(Time1, native, Unit),
+    calendar:system_time_to_rfc3339(Time, [{unit, Unit}, {offset, "z"}]).
+
+%% Returns dt (delta time) in microseconds
+%% @equiv make_dt(A, B, microsecond)
+-spec make_dt(A, B) -> pos_integer() when
+    A :: integer(),
+    B :: integer().
+make_dt(A, B) ->
+    make_dt(A, B, microsecond).
+
+%% Returns monotonic dt (delta time) in specified time_unit()
+-spec make_dt(A, B, Unit) -> pos_integer() when
+    A :: integer(),
+    B :: integer(),
+    Unit :: erlang:time_unit().
+make_dt(A, A, _Unit) when is_integer(A) ->
+    %% Handle edge case when monotonic_time()'s are equal
+    %% Always return a non zero value so we don't divide by zero
+    %% This always returns 1, independent of unit, as that's the smallest
+    %% possible positive integer value delta.
+    1;
+make_dt(A, B, Unit) when is_integer(A) andalso is_integer(B) andalso B > A ->
+    case erlang:convert_time_unit(abs(B - A), native, Unit) of
+        Delta when Delta > 0 ->
+            Delta;
+        _ ->
+            %% Handle case where Delta is smaller than a whole Unit, eg:
+            %% Unit = millisecond,
+            %% (node1@127.0.0.1)2> erlang:convert_time_unit(423, native, Unit).
+            %% 0
+            1
+    end.
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, undefined) ->
+    T;
+add_delta(T, Delta) when is_map(Delta) ->
+    add_delta_int(T, {delta, Delta}).
+
+-spec add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+add_delta_int(T, {delta, _} = Delta) ->
+    {T, Delta}.
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta({Msg, {delta, Delta}}) ->
+    {Msg, Delta};
+extract_delta(Msg) ->
+    {Msg, undefined}.
+
+-spec get_delta(PidRef :: maybe_pid_ref()) -> tagged_delta().
+get_delta(PidRef) ->
+    {delta, make_delta(PidRef)}.
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            maybe_add_delta_int(T, get_delta(get_pid_ref()))
+    end.
+
+%% Allow for externally provided Delta in error handling scenarios
+%% eg in cases like rexi_server:notify_caller/3
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, undefined) ->
+    T;
+maybe_add_delta(T, Delta0) when is_map(Delta0) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            Delta = {delta, Delta0},
+            maybe_add_delta_int(T, Delta)
+    end.
+
+-spec maybe_add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+maybe_add_delta_int(T, {delta, undefined}) ->
+    T;
+maybe_add_delta_int(T, {delta, _} = Delta) ->
+    add_delta_int(T, Delta).
+
+-spec make_delta(PidRef :: maybe_pid_ref()) -> maybe_delta().
+make_delta(undefined) ->
+    undefined;
+make_delta(PidRef) ->
+    TA = get_delta_a(),
+    TB = csrt_server:get_resource(PidRef),
+    Delta = rctx_delta(TA, TB),
+    put_delta_a(TB),
+    Delta.
+
+-spec rctx_delta(TA :: Rctx, TB :: Rctx) -> map().
+rctx_delta(#rctx{} = TA, #rctx{} = TB) ->
+    Delta = #{
+        docs_read => TB#rctx.docs_read - TA#rctx.docs_read,
+        docs_written => TB#rctx.docs_written - TA#rctx.docs_written,
+        js_filter => TB#rctx.js_filter - TA#rctx.js_filter,
+        js_filtered_docs => TB#rctx.js_filtered_docs - TA#rctx.js_filtered_docs,
+        rows_read => TB#rctx.rows_read - TA#rctx.rows_read,
+        changes_returned => TB#rctx.changes_returned - TA#rctx.changes_returned,
+        get_kp_node => TB#rctx.get_kp_node - TA#rctx.get_kp_node,
+        get_kv_node => TB#rctx.get_kv_node - TA#rctx.get_kv_node,
+        db_open => TB#rctx.db_open - TA#rctx.db_open,
+        ioq_calls => TB#rctx.ioq_calls - TA#rctx.ioq_calls,
+        %% "Example to extend CSRT"
+        %% write_kp_node => TB#rctx.write_kp_node - TA#rctx.write_kp_node,
+        %% write_kv_node => TB#rctx.write_kv_node - TA#rctx.write_kv_node,
+        dt => make_dt(TA#rctx.updated_at, TB#rctx.updated_at)
+    },
+    %% TODO: reevaluate this decision
+    %% Only return non zero (and also positive) delta fields
+    %% NOTE: this can result in Delta's of the form #{dt => 1}
+    maps:filter(fun(_K, V) -> V > 0 end, Delta);
+rctx_delta(_, _) ->
+    undefined.
+
+-spec get_delta_a() -> maybe_rctx().
+get_delta_a() ->
+    erlang:get(?DELTA_TA).
+
+-spec put_delta_a(TA :: rctx()) -> maybe_rctx().
+put_delta_a(TA) ->
+    erlang:put(?DELTA_TA, TA).
+
+-spec get_updated_at() -> maybe_integer().
+get_updated_at() ->
+    erlang:get(?LAST_UPDATED).
+
+-spec put_updated_at(Updated :: rctx() | integer()) -> maybe_integer().
+put_updated_at(#rctx{updated_at = Updated}) ->
+    put_updated_at(Updated);
+put_updated_at(Updated) when is_integer(Updated) ->
+    erlang:put(?LAST_UPDATED, Updated).
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    get(?PID_REF).
+
+-spec get_pid_ref(Rctx :: rctx()) -> maybe_pid_ref().
+get_pid_ref(#rctx{pid_ref = PidRef}) ->
+    PidRef;
+get_pid_ref(_) ->
+    undefined.
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    erlang:put(?PID_REF, PidRef),
+    PidRef.
+
+%% @equiv set_fabric_init_p(Func, Enabled, true).
+-spec set_fabric_init_p(Func :: atom(), Enabled :: boolean()) -> ok.
+set_fabric_init_p(Func, Enabled) ->
+    set_fabric_init_p(Func, Enabled, true).
+
+%% Expose Persist for use in test cases outside this module
+-spec set_fabric_init_p(Func, Enabled, Persist) -> ok when
+    Func :: atom(), Enabled :: boolean(), Persist :: boolean().
+set_fabric_init_p(Func, Enabled, Persist) ->
+    Key = fabric_conf_key(Func),
+    ok = config:set_boolean(?CSRT_INIT_P, Key, Enabled, Persist).
+
+-spec fabric_conf_key(Key :: atom()) -> string().
+fabric_conf_key(Key) ->
+    %% Double underscore to separate Mod and Func
+    "fabric_rpc__" ++ atom_to_list(Key).

Review Comment:
   I mentioned this in the top level PR comment, but the use of double underscore is specifically because that's what existing keys in `default.ini` use, there's _zero_ instances in `default.ini` of a key with a semicolon in it, and I had trouble finding clarity on the documentation, so I went with an obviously correct approach.
   
   That said, over in https://github.com/apache/couchdb/pull/5602#discussion_r2221034383 I came up with an alternative approach I'll get wrapped up tomorrow that'll eliminate the need for this function, so I'll delete this soon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236474193


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.
+
+### #rctx.rows_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([fabric_rpc, changes, processed])
+> also Tracking `couch_stats:increment_counter([fabric_rpc, view, rows_read])
+
+A value tracking multiple possible metrics corresponding to rows streamed in
+aggregate operations. This is used for view_rows/changes_rows/all_docs/etc.
+
+### #rctx.changes_returned = 0 :: non_neg_integer() | '_',
+
+The number of `fabric_rpc:changes_row/2` invocations induced by this context,
+specifically tracking the number of changes rows streamed back to the client
+requeest, allowing for distinguishing between the number of changes processed to
+fulfill a request versus the number actually returned in the http response.
+
+### #rctx.ioq_calls = 0 :: non_neg_integer() | '_',
+
+A phony metric counting invocations of `ioq:call/3` induced by this context. As
+with `#rctx.docs_written`, we need a proxy metric to reperesent these calls
+until CSRT context passing is supported so that the `ioq_server` pid and return
+its own delta back to the worker pid.
+
+### #rctx.js_filter = 0 :: non_neg_integer() | '_',
+
+A phony metric counting the number of `couch_query_servers:filter_docs_int/5`
+(eg ddoc_prompt) invocations induced by this context. This is called by way of
+`couch_srt:js_filtered(length(JsonDocs))` which both increments `js_filter` by 1, and
+`js_filtered_docs` by the length of the docs so we can track magnitude of docs
+and doc revs being filtered.
+
+### #rctx.js_filtered_docs = 0 :: non_neg_integer() | '_',
+
+A phony metric counting the quantity of documents filtered by way of
+`couch_query_servers:filter_docs_int/5` (eg ddoc_prompt) invocations induced by
+this context. This is called by way of `couch_srt:js_filtered(length(JsonDocs))`
+which both increments `#rctx.js_filter` by 1, and `#rctx.js_filtered_docs` by
+the length of the docs so we can track magnitude of docs and doc revs being
+filtered.

Review Comment:
   Same thing with https://github.com/apache/couchdb/pull/5602#discussion_r2236455342 CSRT first does coordinator chained to `fabric_rpc` worker, but chaining `fabric_rpc` CSRT contexts into the next process chains will enable tracking of stats at the CSRT context level as its carried out throughout the system.
   
   The delta engine is setup to work for a coordinator getting RPC delta updates, but also for a worker getting couch_file delta updates. Same thing with the coordinator containing a superset of stats for all its workers, the RPC workers would contain a superset of the metrics induced below it, and the same subtraction of any sub process contexts logged can be diffed from the RPC or sup process stats, if reports for those are generated, but we get even further down the rabbit hole of generating lots of reports and it makes me want a Vector/Columnar store at hand hahaha.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236580271


##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->

Review Comment:
   Lol yeah... pros and cons... doing the map pattern guard match inline results in:
   
   > find_matches(Rctxs, #{}=Matchers) when is_list(Rctxs) ->
   
   which now has conflicted uses of guard clauses, but if we try to resolve that inline like with map, we have to do something like:
   
   > find_matches([_|_]=Rctxs, #{}=Matchers) ->
   
   Except now that won't match against `Rctxs=[]` as that's valid value for the type spec `[rctx()]` but neither of those pattern matches are.



##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case couch_srt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = couch_srt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), couch_srt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(Msg, From, St) ->
+    {stop, {unknown_call, Msg, From}, St}.
+
+handle_cast(Msg, St) ->
+    {stop, {unknown_cast, Msg}, St}.
+
+handle_info(restart_config_listener, St) ->
+    ok = subscribe_changes(),
+    {noreply, St};
+handle_info(Msg, St) ->
+    {stop, {unknown_info, Msg}, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbnames_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbnames_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% so we must convert the provided Threshold from milliseconds to native
+    %% representation which then allows us to match against entries where the
+    %% time delta between the started_at and updated_at is greater than the
+    %% native converted threshold.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using couch_srt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = couch_srt_util:tnow(), timer:sleep(10000), T1 = couch_srt_util:tnow(),
+    %%     erlang:convert_time_unit(T1 - T0, native, millisecond).
+    %% 10000
+
+    Unit = millisecond,

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236617644


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   > I could change it to jiffy:json_string(). although I need to check whether compiler would like it or not (the problem might be that jiffy doesn't export its types).
   
   Compiler doesn't care. The dialyzer return warning:
   
   ```
   src/csrt_httpd.erl:90:25: Unknown type jiffy:json_string/0
   src/csrt_httpd.erl:91:23: Unknown type jiffy:some_unknown_type/0
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2229443802


##########
src/couch_srt/README.md:
##########
@@ -206,20 +207,20 @@ The query and HTTP API's are well documented and tested (h/t @iilyak) and
 provide an excellent overview of the interaction patterns and query capabilities
 of CSRT. Those can be found at:
 
-* `csrt_query.erl` "Query API functions"
-  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/csrt_query.erl#L319-L674
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674

Review Comment:
   The sha of this link is most likely need to be updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3100600679

   Thanks for the feedback @iilyak and @nickva, I'll respond inline to the various comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236401327


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.

Review Comment:
   Yeah!! @iilyak did a great job here, if the request specifies a limit larger than the configured limit, an error is returned up front before any work is performed, and in the event a `csrt_query:group_by` groups and aggregates on more than `config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)` rows, that will trigger a `{limit, Acc}` fallback that will bubble up the current aggregations results without further processing. In the event a query is done that returns more than `query_limit()` rows, it automatically initiates a `topK` on those results to constrain the response size to max `query_limit()` groups. Ilya did a great job making sure the data found is funneled back, even in the event of hitting the limits.
   
   I also patched this earlier today (/cc @iilyak) to use `ets:select/3` directly instead of `ets:foldl`, as the former is done directly in the ETS NIF and bypasses a lot of fetching of data into the caller, more details in 711b07073, but I fixed the notion of `query_cardinality_limit` vs `query_limit`, so that `query_limit` is the limit of aggregation groups in the response, and `query_cardinality_limit` is the maximum ets row working set to fetch for handling the query that then attempts to group the data down into less than `query_limit()` groups. Now `csrt_query:group_by` will use `ets:select/3` with `query_cardinality_limit() = 10000 by default` so that the ETS select will happen within the NIF, do the filtering, and then return up to `query_cardinality_limit()` rows which we then aggregate into groups, and run `topK` over with `query_limit()` if need be, while also bubbling up the `limit` hit error to indicate this is a truncated response; the patch also includes a test demonst
 rating hitting the query cardinality limit bubbles this up as well. I config toggled around the existing `ets:foldl` implementation in case folks are interested, but that becomes even worse if we optimize the `ets:select` calls to only return the fields needed in the aggregation as that minimizes the working set size, and can be extended further with special cases like https://www.erlang.org/docs/23/man/ets#select_count-2 to just doing counting (I wish there was a `ets:select_and_agg_by`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236642582


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   The `-dialyzer({no_unknown, [aggregation_result_to_json/2]}).` directive doesn't help.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226795070


##########
src/couch_srt/src/couch_srt.erl:
##########
@@ -0,0 +1,952 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+%% PidRef API
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+
+%% Context Lifecycle API
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+
+%% Public API
+-export([
+    maybe_track_rexi_init_p/1,
+    maybe_track_local_counter/2,
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+
+%% Stats Collection API
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+
+%% RPC API
+-export([
+    rpc_run/1,
+    rpc_unsafe_run/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-opaque query() :: couch_srt_query:query().
+-opaque query_expression() :: couch_srt_query:query_expression().
+-opaque query_option() :: couch_srt_query:query_option().
+
+%%
+%% RPC Operations
+%%
+
+-spec rpc_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, run, [Query])).
+
+-spec rpc_unsafe_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_unsafe_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, unsafe_run, [Query])).
+
+merge_results(Nodes, Resp) ->
+    %% The result of erpc:multicall is returned as a list where the result from each
+    %% node is placed at the same position as the node name is placed in Nodes.
+    %% That is why we can use `lists:zip/2` here.
+    lists:map(fun format_response/1, lists:zip(Nodes, Resp)).
+
+format_response({Node, {ok, {ok, Result}}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {ok, {error, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {ok, Result}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {error, {erpc, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {Tag, _}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Tag]
+    }.
+
+%%
+%% PidRef Operations
+%%
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    couch_srt_util:get_pid_ref().
+
+-spec get_pid_ref(Rctx :: rctx()) -> pid_ref().
+get_pid_ref(Rctx) ->
+    couch_srt_util:get_pid_ref(Rctx).
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    couch_srt_util:set_pid_ref(PidRef).
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    couch_srt_server:create_pid_ref().
+
+-spec destroy_pid_ref() -> maybe_pid_ref().
+destroy_pid_ref() ->
+    destroy_pid_ref(get_pid_ref()).
+
+%%destroy_pid_ref(undefined) ->
+%%    undefined;
+-spec destroy_pid_ref(PidRef :: maybe_pid_ref()) -> maybe_pid_ref().
+destroy_pid_ref(_PidRef) ->
+    erase(?PID_REF).
+
+%%
+%% Context Lifecycle API
+%%
+
+-spec create_worker_context(From, MFA, Nonce) -> pid_ref() | false when
+    From :: pid_ref(), MFA :: mfa(), Nonce :: nonce().
+create_worker_context(From, {M, F, _A}, Nonce) ->
+    case is_enabled() of
+        true ->
+            Type = #rpc_worker{from = From, mod = M, func = F},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_coordinator_context(Httpd, Path) -> pid_ref() | false when
+    Httpd :: #httpd{}, Path :: list().
+create_coordinator_context(#httpd{method = Verb, nonce = Nonce}, Path0) ->
+    case is_enabled() of
+        true ->
+            Path = list_to_binary([$/ | Path0]),
+            Type = #coordinator{method = Verb, path = Path},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_context(Type :: rctx_type(), Nonce :: term()) -> pid_ref() | false.
+create_context(Type, Nonce) ->
+    Rctx = couch_srt_server:new_context(Type, Nonce),
+    PidRef = get_pid_ref(Rctx),
+    set_pid_ref(PidRef),
+    try
+        couch_srt_util:put_delta_a(Rctx),
+        couch_srt_util:put_updated_at(Rctx),
+        couch_srt_server:create_resource(Rctx),
+        couch_srt_logger:track(Rctx),
+        PidRef
+    catch
+        _:_ ->
+            couch_srt_server:destroy_resource(PidRef),
+            %% calling destroy_context(PidRef) clears the tracker too
+            destroy_context(PidRef),
+            false
+    end.
+
+-spec set_context_dbname(DbName :: binary()) -> boolean().
+set_context_dbname(DbName) ->
+    set_context_dbname(DbName, get_pid_ref()).
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: binary(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_dbname(DbName, PidRef).
+
+-spec set_context_handler_fun(Handler) -> boolean() when
+    Handler :: function() | {atom(), atom()}.
+set_context_handler_fun(Handler) ->
+    set_context_handler_fun(Handler, get_pid_ref()).
+
+-spec set_context_handler_fun(Handler, PidRef) -> boolean() when
+    Handler :: function() | {atom(), atom()}, PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun(Fun, PidRef) when is_function(Fun) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            FProps = erlang:fun_info(Fun),
+            Mod = proplists:get_value(module, FProps),
+            Func = proplists:get_value(name, FProps),
+            set_context_handler_fun({Mod, Func}, PidRef)
+    end;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            couch_srt_server:set_context_handler_fun({Mod, Func}, PidRef)
+    end.
+
+%% @equiv set_context_username(User, get_pid_ref())
+set_context_username(User) ->
+    set_context_username(User, get_pid_ref()).
+
+-spec set_context_username(User, PidRef) -> boolean() when
+    User :: null | undefined | #httpd{} | #user_ctx{} | binary(),
+    PidRef :: maybe_pid_ref().
+set_context_username(null, _) ->
+    false;
+set_context_username(_, undefined) ->
+    false;
+set_context_username(#httpd{user_ctx = Ctx}, PidRef) ->
+    set_context_username(Ctx, PidRef);
+set_context_username(#user_ctx{name = Name}, PidRef) ->
+    set_context_username(Name, PidRef);
+set_context_username(UserName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_username(UserName, PidRef).
+
+-spec destroy_context() -> ok.
+destroy_context() ->
+    destroy_context(get_pid_ref()).
+
+-spec destroy_context(PidRef :: maybe_pid_ref()) -> ok.
+destroy_context(undefined) ->
+    ok;
+destroy_context(PidRef) ->
+    %% Stopping the tracker clears the ets entry for PidRef on its way out
+    couch_srt_logger:stop_tracker(),
+    destroy_pid_ref(PidRef),
+    clear_pdict_markers(),
+    ok.
+
+-spec clear_pdict_markers() -> ok.
+clear_pdict_markers() ->
+    ok = lists:foreach(
+        fun
+            ({{csrt, _} = K, _V}) ->
+                erlang:erase(K);
+            (_) ->
+                ok
+        end,
+        erlang:get()
+    ).
+
+%%
+%% Public API
+%%
+
+-spec maybe_track_rexi_init_p({M, F, A}) -> couch_stats:response() when
+    M :: atom(), F :: atom(), A :: non_neg_integer().
+maybe_track_rexi_init_p({M, F, _A}) ->
+    Metric = [M, F, spawned],
+    case couch_srt:should_track_init_p(Metric) of
+        true -> couch_stats:increment_counter(Metric);
+        false -> ok
+    end.
+
+%% Only potentially track positive increments to counters
+-spec maybe_track_local_counter(any(), any()) -> ok.
+maybe_track_local_counter(Name, Val) when is_integer(Val) andalso Val > 0 ->
+    couch_srt:maybe_inc(Name, Val),
+    ok;
+maybe_track_local_counter(_, _) ->
+    ok.
+
+%% @equiv couch_srt_util:is_enabled().
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    couch_srt_util:is_enabled().
+
+%% @equiv couch_srt_util:is_enabled_reporting().
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    couch_srt_util:is_enabled_reporting().
+
+%% @equiv couch_srt_util:is_enabled_rpc_reporting().
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    couch_srt_util:is_enabled_rpc_reporting().
+
+%% @equiv couch_srt_util:is_enabled_init_p().
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    couch_srt_util:is_enabled_init_p().
+
+-spec get_resource() -> maybe_rctx().
+get_resource() ->
+    get_resource(get_pid_ref()).
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(PidRef) ->
+    couch_srt_server:get_resource(PidRef).
+
+%% Log a CSRT report if any filters match
+-spec maybe_report(ReportName :: string(), PidRef :: pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    couch_srt_logger:maybe_report(ReportName, PidRef).
+
+%% Direct report logic skipping should log filters
+-spec do_report(ReportName :: string(), PidRef :: pid_ref()) -> boolean().
+do_report(ReportName, PidRef) ->
+    couch_srt_logger:do_report(ReportName, get_resource(PidRef)).
+
+-spec to_json(Rctx :: maybe_rctx()) -> map() | null.
+to_json(undefined) ->
+    null;
+to_json(Rctx) ->
+    couch_srt_entry:to_json(Rctx).
+
+%%
+%% Stat Collection API
+%%
+
+-spec inc(Key :: rctx_field()) -> non_neg_integer().
+inc(Key) ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key);
+        false ->
+            0
+    end.
+
+-spec inc(Key :: rctx_field(), N :: non_neg_integer()) -> non_neg_integer().
+inc(Key, N) when is_integer(N) andalso N >= 0 ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key, N);
+        false ->
+            0
+    end.
+
+-spec maybe_inc(Stat :: atom(), Val :: non_neg_integer()) -> non_neg_integer().
+maybe_inc(Stat, Val) ->
+    case maps:is_key(Stat, ?STATS_TO_KEYS) of
+        true ->
+            inc(maps:get(Stat, ?STATS_TO_KEYS), Val);
+        false ->
+            0
+    end.
+
+-spec should_track_init_p(Stat :: [atom()]) -> boolean().
+%% "Example to extend CSRT"
+%% should_track_init_p([fabric_rpc, foo, spawned]) ->
+%%    is_enabled_init_p();
+should_track_init_p([fabric_rpc, all_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, changes, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, get_all_security, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, map_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_doc, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_shard, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, reduce_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, update_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p(_Metric) ->
+    false.
+
+-spec ioq_called() -> non_neg_integer().
+ioq_called() ->
+    inc(ioq_calls).
+
+%% we cannot yet use stats couchdb.query_server.*.ddoc_filter because those
+%% are collected in a dedicated process.
+%% TODO: funnel back stats from background worker processes to the RPC worker
+js_filtered(N) ->
+    inc(js_filter),
+    inc(js_filtered_docs, N).
+
+docs_written(N) ->
+    inc(docs_written, N).
+
+-spec accumulate_delta(Delta :: map() | undefined) -> ok.
+accumulate_delta(Delta) when is_map(Delta) ->
+    is_enabled() andalso couch_srt_server:update_counters(get_pid_ref(), Delta),
+    ok;
+accumulate_delta(undefined) ->
+    ok.
+
+-spec make_delta() -> maybe_delta().
+make_delta() ->
+    case is_enabled() of
+        false ->
+            undefined;
+        true ->
+            couch_srt_util:make_delta(get_pid_ref())
+    end.
+
+-spec rctx_delta(TA :: maybe_rctx(), TB :: maybe_rctx()) -> maybe_delta().
+rctx_delta(TA, TB) ->
+    couch_srt_util:rctx_delta(TA, TB).
+
+%%
+%% Aggregate Query API
+%%
+
+-spec active() -> [rctx()].
+active() ->
+    couch_srt_query:active().
+
+-spec active(Type :: json) -> [rctx()].
+active(Type) ->
+    couch_srt_query:active(Type).
+
+-spec active_coordinators() -> [coordinator_rctx()].
+active_coordinators() ->
+    couch_srt_query:active_coordinators().
+
+%% TODO: cleanup json logic here
+-spec active_coordinators(Type :: json) -> [coordinator_rctx()].
+active_coordinators(Type) ->
+    couch_srt_query:active_coordinators(Type).
+
+-spec active_workers() -> [rpc_worker_rctx()].
+active_workers() ->
+    couch_srt_query:active_workers().
+
+-spec active_workers(Type :: json) -> [rpc_worker_rctx()].
+active_workers(Type) ->
+    couch_srt_query:active_workers(Type).
+
+find_by_nonce(Nonce) ->
+    couch_srt_query:find_by_nonce(Nonce).
+
+find_by_pid(Pid) ->
+    couch_srt_query:find_by_pid(Pid).
+
+find_by_pidref(PidRef) ->
+    couch_srt_query:find_by_pidref(PidRef).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_query:find_workers_by_pidref(PidRef).
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    couch_srt_logger:pid_ref_matchspec(AttrName).
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    couch_srt_logger:pid_ref_attrs(AttrName).
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with couch_srt_logger:pid_ref_attrs/1, and
+%% returning on pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    couch_srt_logger:proc_window(AttrName, Num, Time).
+
+-spec query_matcher(MatcherName :: matcher_name()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName) ->
+    couch_srt_query:query_matcher(MatcherName).
+
+-spec query_matcher(MatcherName :: matcher_name(), Limit :: pos_integer()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName, Limit) ->
+    couch_srt_query:query_matcher(MatcherName, Limit).
+
+%%
+%% Delta API
+%%
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, Delta) ->
+    couch_srt_util:add_delta(T, Delta).
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta(T) ->
+    couch_srt_util:extract_delta(T).
+
+-spec get_delta() -> tagged_delta().
+get_delta() ->
+    couch_srt_util:get_delta(get_pid_ref()).
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    couch_srt_util:maybe_add_delta(T).
+
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, Delta) ->
+    couch_srt_util:maybe_add_delta(T, Delta).
+
+%%
+%% Query API functions
+%%
+%%
+
+%% @doc Construct query from the expressions.
+%% There are following types of expressions allowed in the query.
+%%   <li>group_by/1 @see group_by/1</li>
+%%   <li>group_by/2 @see group_by/1</li>
+%%   <li>sort_by/1 @see sort_by/1</li>
+%%   <li>count_by/1 @see count_by/1</li>
+%%   <li>options/1 @see options/1</li>
+%%   <li>from/1 @see from/1</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec query(QueryExpression :: [query_expression()]) ->
+    query() | {error, any()}.
+query(QueryExpression) ->
+    couch_srt_query:query(QueryExpression).
+
+%% @doc Specify the matcher to use for the query.
+%% If atom 'all' is used then all entries would be in the scope of the query.
+%% Also the use of 'all' makes the query 'unsafe'. Because it scans through all entries
+%% and can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    from("docs_read")
+%% ]),
+%% </code>
+%% @end
+-spec from(MatcherNameOrAll :: string() | all) ->
+    query_expression() | {error, any()}.
+from(MatcherNameOrAll) ->
+    couch_srt_query:from(MatcherNameOrAll).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+group_by(AggregationKeys) ->
+    couch_srt_query:group_by(AggregationKeys).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+group_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:group_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+sort_by(AggregationKeys) ->
+    couch_srt_query:sort_by(AggregationKeys).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+sort_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:sort_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'count_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    count_by(username)
+%% ]),
+%% </code>
+%% @end
+-spec count_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+count_by(AggregationKeys) ->
+    couch_srt_query:count_by(AggregationKeys).
+
+%% @doc Construct 'options' query expression.
+%% There are following types of expressions allowed in the query.
+%%   <li>unlimited/0 @see unlimited/0 (cannot be used with 'with_limit/1')</li>
+%%   <li>with_limit/1 @see with_limit/1 (cannot be used with 'unlimited/0')</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      ...
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec options([query_option()]) ->
+    query_expression() | {error, any()}.
+options(OptionsExpression) ->
+    couch_srt_query:options(OptionsExpression).
+
+%% @doc Enable unlimited number of results from the query.
+%% The use of 'unlimited' makes the query 'unsafe'. Because it can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      unlimited()
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec unlimited() ->
+    query_expression().
+unlimited() ->
+    couch_srt_query:unlimited().
+
+%% @doc Set limit on number of results returned from the query.
+%% The construction of the query fail if the 'limit' is greater than
+%% allowed for this cluster.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      with_limit(100)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec with_limit(Limit :: pos_integer()) ->
+    query_expression() | {error, any()}.
+with_limit(Limit) ->
+    couch_srt_query:with_limit(Limit).
+
+%% @doc Executes provided query. Only 'safe' queries can be executed using 'run'.
+%% The query considered 'unsafe' if any of the conditions bellow are met:
+%%   <li>Query uses 'unlimited/0'</li>
+%%   <li>Query uses 'from(all)'</li>
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% run(Q)
+%% </code>
+%% @end
+-spec run(query()) ->
+    {ok, [{aggregation_key(), pos_integer()}]}
+    | {limit, [{aggregation_key(), pos_integer()}]}.
+run(Query) ->

Review Comment:
   At first sight this seems like overkill for an off-to-the-side stats aggregator API meant to be called by a server admin operator to figure out their resource usage. I can see this being needed for a public-facing high rate API like changes or all_docs but here overcomplicates things. Maybe just let the operator pass in a `limit` parameter and have single handler function?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220346513


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   That could make sense in general but I think it can still use some cleanup in this case.
   
    For instance I see
      `{Keys, csrt:query([from(MatcherName), count_by(Keys)])};` 
   but then also
     ` {Keys, query([from(MatcherName), sort_by(Keys, CounterKey)])};` 
   
   *  `couch_util:value(...)` doesn't have the shortened imported format but seems just as verbose as csrt:from(...)
   
   * `with_limit/1` doesn't seem to be used but is imported
   
   There are 17 lines of imports but only 3 lines of `query(...)` calls so it seems overkill in this module as at least?
   
   Sometimes to avoid imports if we're really repeating the same call too often we can make a few short local functions. Say, if we call `erlang:monotonic_time()` then could make local one-line function called `tnow() -> erlang:monotonic_time()` then we can call that 20 or 30 times and it looks a bit better.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221166127


##########
src/couch_stats/src/csrt_entry.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_entry).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-export([
+    value/2,
+    key/1,
+    from_map/1,
+    record_info/0
+]).
+
+%% JSON Conversion API
+-export([
+    convert_type/1,
+    convert_pidref/1,
+    convert_pid/1,
+    convert_ref/1,
+    convert_string/1,
+    to_json/1
+]).
+
+-spec value(rctx_field(), #rctx{}) -> any().
+
+value(pid_ref, #rctx{pid_ref = Val}) -> Val;
+value(nonce, #rctx{nonce = Val}) -> Val;
+value(type, #rctx{type = Val}) -> convert_type(Val);
+value(dbname, #rctx{dbname = Val}) -> Val;
+value(username, #rctx{username = Val}) -> Val;
+value(db_open, #rctx{db_open = Val}) -> Val;
+value(docs_read, #rctx{docs_read = Val}) -> Val;
+value(docs_written, #rctx{docs_written = Val}) -> Val;
+value(rows_read, #rctx{rows_read = Val}) -> Val;
+value(changes_returned, #rctx{changes_returned = Val}) -> Val;
+value(ioq_calls, #rctx{ioq_calls = Val}) -> Val;
+value(js_filter, #rctx{js_filter = Val}) -> Val;
+value(js_filtered_docs, #rctx{js_filtered_docs = Val}) -> Val;
+value(get_kv_node, #rctx{get_kv_node = Val}) -> Val;
+value(get_kp_node, #rctx{get_kp_node = Val}) -> Val;
+value(started_at, #rctx{started_at = Val}) -> Val;
+value(updated_at, #rctx{updated_at = Val}) -> Val.
+
+-spec key(BinKey :: binary() | string() | atom()) ->
+    Key ::
+        rctx_field()
+        | {error, Reason :: any()}.
+
+key(Key) when is_atom(Key) ->
+    key_from_atom(Key);
+key(Key) when is_binary(Key) ->
+    key_from_binary(Key);
+key(Key) when is_list(Key) ->
+    case key_from_binary(list_to_binary(Key)) of
+        {error, {invalid_key, _Key}} ->
+            {error, {invalid_key, Key}};
+        Res ->
+            Res
+    end;
+key(Other) ->
+    key_error(Other).
+
+key_from_atom(pid_ref) -> pid_ref;
+key_from_atom(nonce) -> nonce;
+key_from_atom(type) -> type;
+key_from_atom(dbname) -> dbname;
+key_from_atom(username) -> username;
+key_from_atom(db_open) -> db_open;
+key_from_atom(docs_read) -> docs_read;
+key_from_atom(rows_read) -> rows_read;
+key_from_atom(changes_returned) -> changes_returned;
+key_from_atom(ioq_calls) -> ioq_calls;
+key_from_atom(js_filter) -> js_filter;
+key_from_atom(js_filtered_docs) -> js_filtered_docs;
+key_from_atom(get_kv_node) -> get_kv_node;
+key_from_atom(get_kp_node) -> get_kp_node;

Review Comment:
   I think @iilyak's idea here was to simplify the bubbling up of errors by way of `key_error`, as getting the nested tuple error propagation working properly over RPC and into JSON structures took some finagling, especially when then combined with proper Dialyzer specs.
   
   We have a handful of conversion functions like this as CSRT is predicated on Erlang's ability to efficiently `ets:update_counter` on record fields, but the absence of real records at runtime complicates things.
   
   These functions are somewhat verbose, but verbosity is not an inherent flaw, especially given performance is of key importance here. We started looking into some different options for making these more dynamic by way of exposing the fields using the internal builtin function `record_info/2` that we expose dynamically by way of `csrt_entry:record_info`.
   
   That said, these record fields do not change dynamically, so using runtime reflection here would be an explicit waste of CPU, especially so if we used lists or sets.
   
   I've also thought about some different ideas for compile time macros or parse transforms to build out these types of functions directly, so we don't have the avoided verbosity of re-declaring the fields. That said, verbosity is something that can be well encapsulated by functions, and I think that's done in this case, as the actual caller of `key_from_atom` is:
   
   ```erlang
   -spec key(BinKey :: binary() | string() | atom()) ->
       Key ::
           rctx_field()
           | {error, Reason :: any()}.
   
   key(Key) when is_atom(Key) ->
       key_from_atom(Key);
   key(Key) when is_binary(Key) ->
       key_from_binary(Key);
   key(Key) when is_list(Key) ->
       case key_from_binary(list_to_binary(Key)) of
           {error, {invalid_key, _Key}} ->
               {error, {invalid_key, Key}};
           Res ->
               Res
       end;
   key(Other) ->
       key_error(Other).
   ```
   
   which does a clean job of handling the multicase input being a key or a list of keys, while abstracting over the details of what the valid keys are, but we also still benefit from static analysis here by specifying that the distinct output is one of `rctx_field()` type, which Dialyzer will pickup on discrepancies for.



##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `csrt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `csrt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `csrt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = csrt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = csrt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `csrt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `csrt:set_context_username/{1,2}`.

Review Comment:
   See my comments in https://github.com/apache/couchdb/pull/5602#discussion_r2221121077



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2334514401


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   The `gb_trees` doesn't really fit here. Because it is getting quite expensive to modify on each update. Because each value would have to contain a list of keys corresponding to that value. The update would look something like:
   
   ```
   update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
       Value > Max
   ->
       {[_ | RestOfKeys], SmallestValue, Seq1} = gb_trees:take_smallest(Seq),
       Seq2 = gb_trees:insert(SmallestValue, RestOfKeys, Seq1),
       case gb_trees:take_any(Value, Seq2) of
         {Keys, Seq3} ->
           Top#topK{max = Value, seq = gb_trees:insert(Value, [Key | Keys], Seq3)};
         error ->
           Top#topK{max = Value, seq = gb_trees:insert(Value, [Key], Seq2)};
       end;
   
   ``` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2337330063


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.

Review Comment:
   addressed in ef409f08ef249b2b2f1dbbfae8edcfd26e162a63



##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.

Review Comment:
   addressed in ef409f08ef249b2b2f1dbbfae8edcfd26e162a63



##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->

Review Comment:
   addressed in ef409f08ef249b2b2f1dbbfae8edcfd26e162a63



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316640364


##########
src/couch_srt/README.md:
##########
@@ -206,20 +207,20 @@ The query and HTTP API's are well documented and tested (h/t @iilyak) and
 provide an excellent overview of the interaction patterns and query capabilities
 of CSRT. Those can be found at:
 
-* `csrt_query.erl` "Query API functions"
-  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/csrt_query.erl#L319-L674
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674

Review Comment:
   Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317665376


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling couch_srt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case couch_srt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = couch_srt_util:tnow(),
+            couch_srt_util:put_updated_at(Now),
+            UpdatedInc = couch_srt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(Msg, From, State) ->
+    {stop, {unknown_call, Msg, From}, State}.
+
+handle_cast(Msg, State) ->
+    {stop, {unknown_cast, Msg}, State}.
+
+%%
+%% private functions
+%%
+
+-spec update_element(PidRef :: maybe_pid_ref(), Updates :: [tuple()]) -> boolean().
+update_element(undefined, _Update) ->
+    false;
+update_element({_Pid, _Ref} = PidRef, Update) ->
+    (catch ets:update_element(?CSRT_ETS, PidRef, Update)) == true.

Review Comment:
   Catch `error:badarg` if we're protecting against a missing ets table.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317583142


##########
src/docs/src/config/csrt.rst:
##########
@@ -0,0 +1,587 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. default-domain:: config
+.. highlight:: ini
+
+.. _config-csrt:
+
+==========================================
+Couch Stats Resource Tracker (CSRT) Config
+==========================================
+
+CSRT configuration options and overview.
+
+.. seealso::
+
+    :doc:`/csrt/index`
+
+CSRT config
+===========
+
+This section contains the top level enablement and configuration options for CSRT.
+
+.. config:section:: csrt :: CSRT Primary Configuration
+
+    .. config:option:: enable :: Enable CSRT data collection and RPC deltas
+
+        Core enablement toggle for CSRT, defaults to false. Enabling this
+        setting initiates local CSRT stats collection as well as shipping deltas
+        in RPC responses to accumulate in the coordinator.
+
+        This does *not* trigger the new RPC spawn metrics, and it does not
+        enable reporting for any of the rctx types.
+
+        .. warning::
+
+            You *MUST* have all nodes in the cluster running a CSRT aware
+            CouchDB *before* you enable it on any node, otherwise the old
+            version nodes won't know how to handle the new RPC formats
+            including an embedded Delta payload.
+
+        Top level CSRT enablement for local data collection and RPC deltas::
+
+            [csrt]
+            enable = false
+
+    .. config:option:: enable_init_p :: Enable RPC spawn metric tracking
+
+        Enablement of tracking new metric counters for different `fabric_rpc` operations
+        spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+        inducing database RPC operations within CouchDB, and these init_p metrics aim to
+        provide node level understandings of the workloads being induced by other
+        coordinator processes. This is especially relevant for databases on subsets of a
+        cluster resulting in non-uniform workloads, these metrics are tailored to
+        provide insight into what work is being spawned on each node in the cluster as a
+        function of time.
+
+        Enablement for tracking counts of spawned RPC workers::
+
+            [csrt]
+            enable_init_p = false
+
+    .. config:option:: enable_reporting :: Enable CSRT Process Lifecyle Reports
+
+        This is the primary toggle for enabling CSRT process lifetime reports
+        containing detailed information about the quantity of work induced by
+        the given request/worker/etc. This is the top level toggle for enabling
+        _any_ reporting, and there also exists
+        :config:option:`csrt/enable_rpc_reporting` to disable the reporting of
+        any individual RPC workers, leaving the coordinator responsible of
+        generating a report with the accumulated deltas.
+
+        .. note::
+
+            Note that this setting toggles whether or not to generate process
+            lifecycle reports, but no reports will be generated until logger
+            matchers have been enabled that trigger a match on CSRT contexts
+            that have surpassed the configured thresholds.
+
+        Top level toggle for whether any process lifecycle reports are generated::
+
+            [csrt]
+            enable_reporting = false
+
+    .. config:option:: enable_rpc_reporting :: Enable RPC process lifecyle reports
+
+        This enables the possibility of RPC workers generating reports. They
+        still need to hit the configured thresholds to induce a report, but
+        this will generate CSRT process lifetime reports for individual RPC
+        workers that trigger the configured logger thresholds. This allows for
+        quantifying per node resource usage when desired, as otherwise the
+        reports are at the http request level and don't provide per node stats.
+
+        The key idea here is that having RPC level CSRT process lifetime
+        reporting is incredibly useful, but can also generate large quantities
+        of data. For example, a view query on a Q=64 database will stream
+        results from 64 shard replicas, resulting in at least 64 RPC reports,
+        plus any that might have been generated from RPC workers that "lost"
+        the race for shard replica. This is very useful, but a lot of data
+        given the verbose nature of funneling it through the RSyslog reports,
+        however, the ability to write directly to something like ClickHouse or
+        another columnar store would be great.
+
+        Until there's an efficient storage mechanism to stream the results to,
+        the rsyslog entries work great and are very practical, but care must be
+        taken to not generate too much data for aggregate queries as they
+        generate at least `Qx` more report than an individual report per http
+        request from the coordinator.  This setting exists as a way to either
+        a) utilize the logger matcher configured thresholds to allow for _any_
+        rctx's to be recorded when they induce heavy operations, either
+        Coordinator or RPC worker; or b) to _only_ log workloads at the
+        coordinator level.
+
+        .. note::
+
+            This setting exists because we lack an expressive enough config
+            declaration to easily chain the matchspec constructions as
+            `ets:fun2ms/1` is a special compile time parse transform macro that
+            requires the full definition to be specified directly, it cannot
+            be interactively constructed. That said, you _can_ register matchers
+            through `remsh` with more specific and fine grained pattern matching,
+            and a more expressive system for defining matchers are being
+            explored.
+
+        .. warning::
+
+            Enabling this setting *will* generate considerably more logs! Specifically, for aggregate queries and database operations, this will generate `Q` * `N` times more logs than a singular doc request taking only `N` inreacting with a singular shard range. See the note above about this being a temporary setting during the experimental stages of CSRT.
+
+        Toggle to enable possibility of RPC process lifecycle reports::
+
+            [csrt]
+            enable_rpc_reporting = false
+
+    .. config:option:: should_truncate_reports :: truncate zero values from lifecyle reports
+
+        enables truncation of the csrt process lifetime reports to not include
+        any fields that are zero at the end of process lifetime, eg don't
+        include `js_filter=0` in the report if the request did not induce
+        javascript filtering.
+
+        this can be disabled if you really care about consistent fields in the
+        report logs, but this is a log space saving mechanism, similar to
+        disabling rpc reporting by default, as its a simple way to reduce
+        overall volume
+
+        Truncate zero values from process lifecycle reports, enabled by default:
+
+            [csrt]
+            should_truncate_reports = true
+
+    .. config:option:: query_limit :: Maximum quantity of rows to return in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_limit = 100
+
+    .. config:option:: query_cardinality_limit :: Maximum quantity of rows to allow in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_cardinality_limit = 10000
+
+.. _csrt-logger-matcher-configuration:
+
+CSRT Logger Matcher Configuration
+=================================
+
+There are currently eight builtin default logger matchers designed to make it
+easy to do filtering on heavy resource usage inducing and long running
+requests. These are designed as a simple baseline of useful matchers, declared
+in a manner amenable to `default.ini` based constructs. More expressive matcher

Review Comment:
   If we're going for a monospace code font then we'd need  a double-backtick otherwise it looks italicized (which is not bad, but it depends what we're aiming for):
   
   <img width="585" height="59" alt="Screenshot 2025-09-02 at 10 09 01 PM" src="https://github.com/user-attachments/assets/9187d981-f758-43dc-ad83-284bd65bd151" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236441597


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.

Review Comment:
   The process lifecycle reports use the local node's `erlang:monotonic_time()` representation, so all of the process lifecycle reports happen on the local node. The RPC workers stream their stats directly to the coordinator, so there's no final cleanup delta to send, the worker dies and we either log or we don't, but the data has already been sent to the coordinator.
   
   If any additional RPC worker reports are logged in conjunction with a coordinator report, subtracting those values from the final coordinator stats will eliminate the duplication of field counting in the reports.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3129079209

   Thanks for the additional feedback Ilya, I've fixed up all the typos in 6481338c6, much appreciated.
   
   Fwiw I'm having a lot of trouble seeing the commit specific comments, I didn't see those in the PR at all, I happened to notice an email notification. For some reason, those conversations are not showing up for me in this top level PR, or the specific PR commit https://github.com/apache/couchdb/pull/5602/commits/828667e3d724c43763ce772180b748287b8c4b8c but I finally found your comments over in the raw commit page at https://github.com/apache/couchdb/commit/828667e3d724c43763ce772180b748287b8c4b8c .
   
   I've sorted out all of those comments aside from the recommendation to switch to Erlang code blocks, as that made Sphinx not happy, oddly it complained about the `#` in a `#Ref<x.y.z>` that output in remsh... kind of odd to not to recognize that for Erlang, although it's _technically_ not the proper language you'd think it would handle console output from that language haha.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236558400


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   Yeah, it's `erlang:string()`, there's a number of things flowing around as `string()` for better or worse, but on a positive note the type_specs greatly simplify making changes and testing validity of the conversions. Perhaps @iilyak can highlight the specifics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236536174


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   I see now that it is confusing. Here is how I came about this spec. The query_result function return JSON as expected by `jiffy:encode/1` (`jiffy:json_value()`). However `jiffy:json_value()` is opaque and cannot serve as a documentation. So my first idea was to have a comment which shows the structure of the result. However maintaining the comment is harder because compiler doesn't check the comments. So I invented `-type json_spec(_Spec) :: term().` This type *ignores* what's inside. It is meant for documenting JSON structure. I thought it would be great to use across codebase to provide example of the data structures used by functions. So I used the types which make sense to document JSON. I.e. `string()` in this context refer to the JSON string. I could change it to `jiffy:json_string().` although I need to check whether compiler would like it or not (the problem might be that `jiffy` doesn't export its types).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2323550276


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->

Review Comment:
   I am not sure if that can happen for ets:lookup?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317578906


##########
src/docs/src/csrt/index.rst:
##########
@@ -0,0 +1,541 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. _csrt:
+
+===================================
+Couch Stats Resource Tracker (CSRT)
+===================================
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with
+the sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces
+those metrics inc calls, and then CSRT updates an ets entry containing the
+context information for the local process, such that global aggregate queries
+can be performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecycle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable approach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occur in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well,
+with zero concurrent writes from any other processes. Outside of the context
+setup and teardown logic, _only_ operations to `ets:update_counter` are
+performed, one per process invocation of `couch_stats:increment_counter`, and
+one for coordinators to update worker deltas in a single batch, resulting in a
+1:1 ratio of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework itself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in retroactively
+understanding the quantity of resources induced by a query like
+`/db/_changes?since=$SEQ`, or similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we
+should instead use the existing stats tracking mechanisms that have already
+been deemed critical information to track, which then also allows for the real
+time tracking and aggregate query capabilities. The Mango execution stats can
+be ported into CSRT itself and just become one subset of the stats tracked as a
+whole, and similarly, any additional desired stats tracking can be easily added
+and will be picked up in the RPC deltas and process lifetime reports.
+
+.. seealso::
+
+    :doc:`/config/csrt`
+
+CSRT Overview
+-------------
+
+    When an incoming HTTP request is is handled by the CouchDB `chttpd` worker

Review Comment:
   `is is` -> `is`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316520697


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.

Review Comment:
   That makes sense thank you for explaining



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2337332822


##########
src/couch_srt/src/couch_srt_util.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+-include_lib("couch_srt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    config:get_boolean(?CSRT, "enable", false).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    config:get_boolean(?CSRT, "enable_init_p", false).
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    config:get_boolean(?CSRT, "enable_reporting", false).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking

Review Comment:
   addressed in 07754362a55880eab509fc322a93fe144a442bad



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2347106937


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   Wonder why we would need the have a list of keys for each value separately,  maybe we can keep them together?  Maybe something like this could work:
   
   ```erlang
   topk(#{} = Map, T) when is_integer(T), T > 0 ->
       Fun = fun(K, V, Set0) ->
           Set1 = gb_sets:add({V, K}, Set0),
           case gb_sets:size(Set1) > T of
               true -> element(2, gb_sets:take_smallest(Set1));
               false -> Set1
           end
        end,
       Set = maps:fold(Fun, gb_sets:empty(), Map),
       lists:reverse([{K, V} || {V, K} <- gb_sets:to_list(Set)]).
   ```
   
   ```
   > Map = #{a => 1.0, b => 1.0, c => 0.5, d => 0.75, e => 1.2, f => 1.2, g => 1.1, h => 3.8}.
   
   > lists:foreach(fun(K) ->
       io:format(" topK ~p : ~p ~n", [K, topk:topk(Map, K)])
      end, lists:seq(1, 9)).
   
    topK 1 : [{h,3.8}]
    topK 2 : [{h,3.8},{f,1.2}]
    topK 3 : [{h,3.8},{f,1.2},{e,1.2}]
    topK 4 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1}]
    topK 5 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1},{b,1.0}]
    topK 6 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1},{b,1.0},{a,1.0}]
    topK 7 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1},{b,1.0},{a,1.0},{d,0.75}]
    topK 8 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1},{b,1.0},{a,1.0},{d,0.75},{c,0.5}]
    topK 9 : [{h,3.8},{f,1.2},{e,1.2},{g,1.1},{b,1.0},{a,1.0},{d,0.75},{c,0.5}]
   ```
   
   It should be a bit smaller and have better complexity `O(n log k)` vs previous `O (n * k log k)` (I think?) since we're not resorting the top k list on every insertion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2352519777


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   Thank you Nick!!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349783372


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   > I would even skip the whole update_topK function and the record and just use the topk from above as is. 
   
   The `update_topK` skips the update when `Value < Min`, we would have many small values so we could safe a lot of updates.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2217005564


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([

Review Comment:
   These two functions are no longer available. The replacement functions are
   
   ```erlang
       rpc_run/1,
       rpc_unsafe_run/1
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220346513


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   That could make sense in general but I think it can still use some cleanup in this case.
   
    For instance I see
      `{Keys, csrt:query([from(MatcherName), count_by(Keys)])};` 
   but then also
     ` {Keys, query([from(MatcherName), sort_by(Keys, CounterKey)])};` 
   
   *  `couch_util:value(...)` doesn't have the shorted imported format but seems just as verbose as csrt:from(...)
   
   * `with_limit/1` doesn't seem to be used but is imported
   
   There 17 lines of imports but only 3 lines of `query(...)` calls so it seems overkill in this module as at least?
   
   Sometimes to avoid imports if we're really repeating the same call too often we can make a few short local functions. Say, if we call `erlang:monotonic_time()` then could make local one-line function called `tnow() -> erlang:monotonic_time()` then we can call that 20 or 30 times and it looks a bit better.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221104996


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.

Review Comment:
   I've elaborated on this more in the CSRT.md docs, but basically, the point of `csrt:is_enabled_rpc_reporting` is to give the database user the choice of whether to have thresholds triggering off of _any_ coordinator or rpc worker context that hits the threshold, or just the coordinators. This is all for the purposes of constraining log volume, as if we had an efficient binary columnar store I'd just write _all_ context reports to that and we've had 100% coverage, but the fact that view queries and other aggregate queries balloon out aggressively on RPC worker inducement as a function of Q combined with these logs currently being designed for RSyslog ascii text means that we need to cautious with log volumes.
   
   These settings and the default logger matchers and the dbname_io matchers are just pragmatic ways to try and make it easy for users to enable a subset of report logging for what they need, when and where they need it, and I think CSRT genuinely is a paradigm shift on that front, as being able to do something like:
   
   > rpc:multicall(config, set, ["csrt_logger.dbnames_io", "foo", "12345"]).
   
   And dynamically enable reporting on the cluster for all http requests to db `foo` that induce more than `12345` of any particular IO operation is a phenomenal improvement in introspection, let alone being able to enable it and query it in real time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226765465


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.

Review Comment:
   Hmm, maybe we should make it a real metric: fabric_doc_written or something? Would it tell us anything interesting or would be pretty much identical to the one db updater it would be just a waste...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3105477204

   Over in 38a53a059 I migrated CSRT into the `couch_srt` application, aka CSRT for short. For context, I did so with a series of sed commands and a bit of manual cleanup. I did that incrementally and did local git commits with the different steps and sed commands as commit messages where used, which I then squashed down into a final commit before pushing out here, but here's the squashed sequence of steps to generate the above git sha:
   
   ```
   Extract CSRT into dedicated couch_srt application
   sed -I '' 's/csrt.hrl/couch_srt.hrl/g' src/*/{src,test/eunit}/*.erl
   
   sed -I '' 's/csrt:/couch_srt:/g' src/*/{src,test/eunit}/*.erl
   
   sed -I '' 's/csrt_\([a-z_]*\):/couch_srt_\1:/g' src/*/{src,test/eunit}/*.erl
   
   Cleanup remaining 'csrt\(_[a-z_]*\)\?:' references
   
   Hook in couch_srt app
   
   sed -I '' 's/^-module(csrt/-module(couch_srt/g' src/*/{src,test/eunit}/*.erl
   
   More cleanup
   ```
   
   That commit creates a clean separation between `couch_stats` and `couch_srt`, and I even moved most of the `couch_stats` changes to within `couch_srt`.
   
   One thing that I'd like feedback on in particular, I preserved the name "CSRT" as I think it's a good name for the system and I've grown accustomed to referring to it as such. The public Erlang APIs all use `couch_srt`, but internally within I've kept the use of `csrt` in a number of places. I still think CSRT is a better and more concise name than `couch_srt`, so I've left that in the documentation and config settings as I think the conciseness of "csrt" helps in a number of places, for instance, I'm still using `"csrt"` and `"csrt_logger.matchers_threshold"` for the config settings as I think that's cleaner and more user friendly that `couch_srt_logger.matchers_threshold`.
   
   I like the balance of `"couch_srt"` and `"csrt"` in that commit so I didn't go further on renaming things in `couch_srt.hrl`, but let me know what y'all think.
   
   I kept that as an isolated commit to have a logical commit containing all the direct sed changes, and then I waited to run `make erlfmt-format` until afterwards so folks could reasonably follow the sed commit and not be surprised by the format updates in 86ba96be0.
   
   Tomorrow I'll add some additional updates to the overview documentation to better illustrate when/where/how users would initially enable and utilize CSRT.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2237642682


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   @iilyak fwiw, I've been seeing different errors out of Dialyzer since switching to OTP 26, I haven't had a chance to track it down but those Unknown type errors I suspect are related to Dialyzer not properly loading in the PLTs or something. For example, it seems to have lost all connections with Erlang applications outside the immediate application, eg:
   
   ```
   src/couch_srt.erl:146:13: Unknown function mem3:nodes/0                                                                                                                                                                                              
   src/couch_srt.erl:158:13: Unknown function mem3:nodes/0                                                                   
   src/couch_srt.erl:360:45: Unknown type couch_stats:response/0                                                             
   src/couch_srt_httpd.erl:31:5: Unknown function chttpd:validate_ctype/2                                                    
   src/couch_srt_httpd.erl:32:19: Unknown function chttpd:json_body_obj/1                                                                                                                                                                               
   src/couch_srt_httpd.erl:51:30: Unknown function couch_util:get_value/2                                                    
   src/couch_srt_httpd.erl:64:13: Unknown function chttpd:send_json/2                                                        
   src/couch_srt_httpd.erl:70:10: Unknown function chttpd:verify_is_server_admin/1                                           
   ```
   
   but those functions are fine, like `mem3:nodes()` lol. Something odd with the Dialyzer configs, I thought it had gotten some new updates around PLTs, @iilyak I seem to remember you mentioning OTP 26 had an optimization to reuse the PLTs or something? It sure seems like we're not loading and generating them properly now that we're on OTP 26, maybe we need to update some of the Dialyzer invocations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236446852


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```

Review Comment:
   The `clear_pdict_makers` call, no, but the rest yes, so I updated the documentation explaining when to use those functions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2350010924


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   ```erlang
   topk(#{} = Map, T) when is_integer(T), T > 0 ->
       Fun = fun(K, V, Set0) ->
           case gb_sets:size(Set0) >= T of
               true ->
                   case V =< element(1, gb_sets:smallest(Set0)) of
                       true -> Set0;
                       false -> element(2, gb_sets:take_smallest(gb_sets:add({V, K}, Set0)))
                   end;
               false ->
                   gb_sets:add({V, K}, Set0)
           end
       end,
       Set = maps:fold(Fun, gb_sets:empty(), Map),
       lists:reverse([{K, V} || {V, K} <- gb_sets:to_list(Set)]).
   ```
   
   This is a more optimized version ^
   
   The savings when getting top 100 items with various randomly generated maps of kv:
     * 11 -> 3 msec for 10k kvs
     * 95 -> 22 msec for 100k kvs
     * 862 -> 166 msec for 1m kvs
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316628965


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   > Yeah, it's erlang:string(), there's a number of things flowing around as string() for better or worse, but on a positive note the type_specs greatly simplify making changes and testing validity of the conversions.
   
   This is a case of better not have any type specs than have wrong type specs. `erlang:string()` is essentially a list and becomes an array in json. It doesn't make sense to have usernames in json as arrays, they would look something like `["u","s","e","r","1"]` which is silly
   
   > I added the json_string() type.
   
   That works, thanks!
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317666856


##########
src/couch_srt/src/couch_srt_util.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+-include_lib("couch_srt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    config:get_boolean(?CSRT, "enable", false).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    config:get_boolean(?CSRT, "enable_init_p", false).
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    config:get_boolean(?CSRT, "enable_reporting", false).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking

Review Comment:
   Monotonic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316522276


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316515583


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.

Review Comment:
   > @nickva do you think there's any core data structure issues with our ets #rctx{} setup that would prevent this from being added later? Personally I don't think so, I think CSRT is in a good spot with solid data collection and a well tested/documented/type_spec'ed HTTP and query API built on the performant matchers, and it's in a good spot to start putting to use and figure out what types of additional querying and introspection capabilities are needed.
   
   I agree, I think it can be added later and everything is already there. However, if that is the main/primary usage of the PR (show me what induces most load on my nodes) the overall design could be simpler.  I would have done something like keep a set of ets tables per node, one per every 5 seconds and maybe keep 12 total to get per minute aggregated. Then we could emit reports starting with every 5 seconds and up to a minute and avoid altering all the rexi message and avoid some of the configuration complexity. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317657994


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->

Review Comment:
   this should probably be error:badarg only, what other exceptions would we catch here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2323405395


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   The maximum length of the list is equal to capacity, which is equal to number of entries we want to return. This meant to be small (in order of 5-20, maybe 100 at maximum). For small number of elements the list would probably be faster. However I like an idea to use gb_tree. Especially because I wouldn't have to maintain `size`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221252151


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([

Review Comment:
   Thanks, fixed up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221227453


##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).

Review Comment:
   Kind of... this is the alternative to using `csrt:is_enabled_rpc_reporting`, whereas we can easily filter on a coordinator here, but it's tricky to conditionally generate those functions within the parens of the parse transform. I started working on directly generated the matchspecs, but did not complete a proper version yet.
   
   I'll delete these.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221064573


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]

Review Comment:
   I've added additional documentation for this over in CSRT.md at `## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").` section, and similarly `# CSRT Logger Matcher Enablement and Thresholds ` for the default matchers.
   
   You can specify a dbname or a shard name, and that will match at either the coordinator context or the RPC worker context, respectively, so if you match on db `foo`, that is a clustered dbname that exists at the coordinator level, so it'll match coordinators for requests against db `foo`, whereas RPC workers operate on the db shards themselves, so you'd specify `shards/80000000-ffffffff/foo.1744078714` to match on a particular RPC worker, but you also need `csrt:is_enabled_rpc_reporting()` to trigger logs for RPC workers. The idea is until a properly expressive config syntax is defined for mapping complex matchers to default.ini, we instead have the ability to easily set specific target thresholds on the dimensions and then specify whether or not to enable RPC reporting.
   
   As it says in the comment, this will trigger on any _one_ field being greater than the threshold. Again, this is address the expressiveness of the ini settings and what we can map easily into `ets:fun2ms`: this is a `key => val` config lookup on a dbname to an integer threshold as a simple way to register a heavy IO matcher against a database. Ideally, this becomes obsolete with more expressive matchers, but pragmatically it's a very useful way to be able to config:set a logger matcher dynamically and get useful data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2219990347


##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format

Review Comment:
   Maybe a clarification needed, the `Time0` argument we get is not in milliseconds but in `native` units right? And it's the monotonic timestamp from the same node? So we add the offset + Time0 in native then convert it to rfc3339 with 3 digits millisecond precision.
   
   I think something like this one-liner could work then:
   
   ```erlang
   calendar:system_time_to_rfc3339(Time0 + erlang:time_offset(), [{unit, native}, {offset, "z"}]).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3101170133

   Thanks @iilyak and @nickva, I've addressed most of the comments in 2357812d4 as well as CSRT.md updates in debe52fab addressing Ilya's comments and also adding an updated overview, list of examples, and links to more info.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349783372


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   > I would even skip the whole update_topK function and the record and just use the topk from above as is. 
   
   The `update_topK` skips the update when `Value < Min`, we would have many small values so we could safe a lot of updates.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak-ibm (via GitHub)" <gi...@apache.org>.
iilyak-ibm commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2337330843


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->

Review Comment:
   addressed in ef409f08ef249b2b2f1dbbfae8edcfd26e162a63



##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling couch_srt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case couch_srt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = couch_srt_util:tnow(),
+            couch_srt_util:put_updated_at(Now),
+            UpdatedInc = couch_srt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(Msg, From, State) ->
+    {stop, {unknown_call, Msg, From}, State}.
+
+handle_cast(Msg, State) ->
+    {stop, {unknown_cast, Msg}, State}.
+
+%%
+%% private functions
+%%
+
+-spec update_element(PidRef :: maybe_pid_ref(), Updates :: [tuple()]) -> boolean().
+update_element(undefined, _Update) ->
+    false;
+update_element({_Pid, _Ref} = PidRef, Update) ->
+    (catch ets:update_element(?CSRT_ETS, PidRef, Update)) == true.

Review Comment:
   addressed in ef409f08ef249b2b2f1dbbfae8edcfd26e162a63



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317580110


##########
src/docs/src/config/csrt.rst:
##########
@@ -0,0 +1,587 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. default-domain:: config
+.. highlight:: ini
+
+.. _config-csrt:
+
+==========================================
+Couch Stats Resource Tracker (CSRT) Config
+==========================================
+
+CSRT configuration options and overview.
+
+.. seealso::
+
+    :doc:`/csrt/index`
+
+CSRT config
+===========
+
+This section contains the top level enablement and configuration options for CSRT.
+
+.. config:section:: csrt :: CSRT Primary Configuration
+
+    .. config:option:: enable :: Enable CSRT data collection and RPC deltas
+
+        Core enablement toggle for CSRT, defaults to false. Enabling this
+        setting initiates local CSRT stats collection as well as shipping deltas
+        in RPC responses to accumulate in the coordinator.
+
+        This does *not* trigger the new RPC spawn metrics, and it does not
+        enable reporting for any of the rctx types.
+
+        .. warning::
+
+            You *MUST* have all nodes in the cluster running a CSRT aware
+            CouchDB *before* you enable it on any node, otherwise the old
+            version nodes won't know how to handle the new RPC formats
+            including an embedded Delta payload.
+
+        Top level CSRT enablement for local data collection and RPC deltas::
+
+            [csrt]
+            enable = false
+
+    .. config:option:: enable_init_p :: Enable RPC spawn metric tracking
+
+        Enablement of tracking new metric counters for different `fabric_rpc` operations
+        spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+        inducing database RPC operations within CouchDB, and these init_p metrics aim to
+        provide node level understandings of the workloads being induced by other
+        coordinator processes. This is especially relevant for databases on subsets of a
+        cluster resulting in non-uniform workloads, these metrics are tailored to
+        provide insight into what work is being spawned on each node in the cluster as a
+        function of time.
+
+        Enablement for tracking counts of spawned RPC workers::
+
+            [csrt]
+            enable_init_p = false
+
+    .. config:option:: enable_reporting :: Enable CSRT Process Lifecyle Reports
+
+        This is the primary toggle for enabling CSRT process lifetime reports
+        containing detailed information about the quantity of work induced by
+        the given request/worker/etc. This is the top level toggle for enabling
+        _any_ reporting, and there also exists
+        :config:option:`csrt/enable_rpc_reporting` to disable the reporting of
+        any individual RPC workers, leaving the coordinator responsible of
+        generating a report with the accumulated deltas.
+
+        .. note::
+
+            Note that this setting toggles whether or not to generate process
+            lifecycle reports, but no reports will be generated until logger
+            matchers have been enabled that trigger a match on CSRT contexts
+            that have surpassed the configured thresholds.
+
+        Top level toggle for whether any process lifecycle reports are generated::
+
+            [csrt]
+            enable_reporting = false
+
+    .. config:option:: enable_rpc_reporting :: Enable RPC process lifecyle reports
+
+        This enables the possibility of RPC workers generating reports. They
+        still need to hit the configured thresholds to induce a report, but
+        this will generate CSRT process lifetime reports for individual RPC
+        workers that trigger the configured logger thresholds. This allows for
+        quantifying per node resource usage when desired, as otherwise the
+        reports are at the http request level and don't provide per node stats.
+
+        The key idea here is that having RPC level CSRT process lifetime
+        reporting is incredibly useful, but can also generate large quantities
+        of data. For example, a view query on a Q=64 database will stream
+        results from 64 shard replicas, resulting in at least 64 RPC reports,
+        plus any that might have been generated from RPC workers that "lost"
+        the race for shard replica. This is very useful, but a lot of data
+        given the verbose nature of funneling it through the RSyslog reports,
+        however, the ability to write directly to something like ClickHouse or
+        another columnar store would be great.
+
+        Until there's an efficient storage mechanism to stream the results to,
+        the rsyslog entries work great and are very practical, but care must be
+        taken to not generate too much data for aggregate queries as they
+        generate at least `Qx` more report than an individual report per http
+        request from the coordinator.  This setting exists as a way to either
+        a) utilize the logger matcher configured thresholds to allow for _any_
+        rctx's to be recorded when they induce heavy operations, either
+        Coordinator or RPC worker; or b) to _only_ log workloads at the
+        coordinator level.
+
+        .. note::
+
+            This setting exists because we lack an expressive enough config
+            declaration to easily chain the matchspec constructions as
+            `ets:fun2ms/1` is a special compile time parse transform macro that
+            requires the full definition to be specified directly, it cannot
+            be interactively constructed. That said, you _can_ register matchers
+            through `remsh` with more specific and fine grained pattern matching,
+            and a more expressive system for defining matchers are being
+            explored.
+
+        .. warning::
+
+            Enabling this setting *will* generate considerably more logs! Specifically, for aggregate queries and database operations, this will generate `Q` * `N` times more logs than a singular doc request taking only `N` inreacting with a singular shard range. See the note above about this being a temporary setting during the experimental stages of CSRT.
+
+        Toggle to enable possibility of RPC process lifecycle reports::
+
+            [csrt]
+            enable_rpc_reporting = false
+
+    .. config:option:: should_truncate_reports :: truncate zero values from lifecyle reports
+
+        enables truncation of the csrt process lifetime reports to not include
+        any fields that are zero at the end of process lifetime, eg don't
+        include `js_filter=0` in the report if the request did not induce
+        javascript filtering.
+
+        this can be disabled if you really care about consistent fields in the
+        report logs, but this is a log space saving mechanism, similar to
+        disabling rpc reporting by default, as its a simple way to reduce
+        overall volume
+
+        Truncate zero values from process lifecycle reports, enabled by default:

Review Comment:
   Add another `:` so it's rendered like a config snippet.  Currently it looks like:
   
   <img width="678" height="226" alt="Screenshot 2025-09-02 at 10 06 01 PM" src="https://github.com/user-attachments/assets/8d4c3439-8807-4379-a1c0-881959fae7fd" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3101278621

   Alright I went ahead and reworked the `should_track_init_p` logic in d757adda0 to no longer use the dynamic config lookups and instead use direct function head definitions on the desired metrics. This deletes the `[csrt.init_p]` default.ini section completely and entirely  This has the added benefit of removing the fields from the `default.ini` section using the `__` separator, avoiding that issue entirely, although on that front, it'd be worthwhile to formally declare what types of values are acceptable in the keys, as I've been rather hesitant to go wild on that front.
   
   This also removed the ugly string conversion in `csrt_util:fabric_conf_key`, and as much emphasis I've made in this PR on avoiding dynamic runtime string conversions for expected entities, I'm very much glad to delete the ugliness of that string conversion! 😅 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223244564


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `csrt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```

Review Comment:
   I don't think time handling is that confusing and it's documented in the erlang docs, so a link to the function or docs there should be good enough. A short comment in the function is fine too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3104645567

   I talked with @nickva out of band and realized I confused myself on the need for using absolute values around the time deltas for negative timestamps, negative negative numbers confused me, but he's correct that the use of `abs` there is extraneous and can be removed. I'll take care of that and he also convinced me to extract out CSRT from couch_stats into a dedicated application, so I'll sort that out now as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226699616


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.

Review Comment:
   That's why I was thinking maybe having a per/node per/time window summary mode might help to avoid the logs being overloaded and still get the rpc-level data. Kind of a middle ground or that works out could be just the default (one) mode so we (users) won't have to choose between only vs coordinator + enable_rpc_reporting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226806746


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   Is this the erlang `string()` type or our own locally defined version. I don't remember types very well but looking it up I see `-type string() :: [[char]]` which a list of character so if we emit it as with jiffy won't it be encoded as an array?
   
   ```
   jiffy:encode(#{value => [$a, $b, $c]}).
   <<"{\"value\":[97,98,99]}">>
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236175785


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need

Review Comment:
   The thresholds apply to any `rctx()`, we have a safety hatch to disable RPC reporting entirely, by way of:
   
   ```erlang
   %% Return a subset of Matchers for each Matcher that matches on Rctxs
   -spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
   find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
       Rctxs1 =
           case couch_srt_util:is_enabled_rpc_reporting() of
               true ->
                   Rctxs;
               false ->
                   [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
           end,
       maps:filter(
           fun(_Name, {_MSpec, CompMSpec}) ->
               (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
           end,
           Matchers
       ).
   ```
   
   All process lifecycle reports happen on the local node, but the RPC workers can funnel their stats to the remote coordinator process, technically `The thresholds here are per node per worker` is `The thresholds here are per node per coordinator or RPC worker context`, eg if you enable the `ioq_calls` matcher with `Threshold=10000`, and you also enable RPC reporting, then any coordinator or RPC worker inducing 10000 IOQ calls will log, which naturally means the coordinator would also log given it had a worker alone that induced 10000 IOQ calls. It's a bit awkward, but it's a pragmatic escape hatch if you need to enable RPC report logging on a node or cluster.
   
   Over in 828667e3d I've reworked the docs extensively into the Sphinx docs app, and I've covered details on enabling RPC reporting in both the main CSRT docs index and the CSRT config index, with details covering the nuances. I don't know how to link out to branch level built Sphinx documentation, but I put an entire section together highlighting these nuances, why it's a problem, why it's a temporary problem, and how to specify the clustered db name vs the sharded db name in the RPC workers, over in http://localhost:8000/csrt/index.html#demonstration-of-expressiveness-constraints-in-logger-matchers-and-ini-settings (that link should work if you run it locally, or eventually by changing the host on the documentation).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236291617


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.

Review Comment:
   Yeah I agree that would be nice, it's just not immediately obvious to me what that should look like. I've mentioned up above in https://github.com/apache/couchdb/pull/5602#discussion_r2236220507 some of the issues around large `Q` databases and the potential RPC volume.
   
   To be clear, I agree this would be nice, but I think this is a secondary phase type of thing, as this kind of strikes like me setting up agents that periodically query to periodically perform an aggregation and log the results. It also doesn't seem like a data structure issue on the core CSRT tracking, but rather another instance of the expressiveness problem with the matchers and the ini settings.
   
   I mentioned above reworking the CSRT docs considerably and moving them into the Sphinx docs page, but I kept the old `CSRT.md` as `couch_srt/README.md`, and I've updated it based on your feedback, but I also included a section on `# Extending CSRT` and `## Next Steps`, where I mention adding report "snapshots".
   
   I've always kind of though of report "snapshots" as a natural evolution to the process lifecyle reports, especially for long running background jobs once those are support. For instance, it'd be great to get periodic progress status reports generated for large index builds.
   
   @nickva do you think there's any core data structure issues with our ets `#rctx{}` setup that would prevent this from being added later? Personally I don't think so, I think CSRT is in a good spot with solid data collection and a well tested/documented/type_spec'ed HTTP and query API built on the performant matchers, and it's in a good spot to start putting to use and figure out what types of additional querying and introspection capabilities are needed.
   
   For instance, if we had periodic per node processes logging these stats, what would they aggregate on? We could certainly do something like `generate a report on top 10 IOQ calls grouped by user` or something, but we lose a lot of the nuance in the data there, making it harder to work with the aggregated report values. Another idea I had was during the context destruction when we potentially generate a report, we have the final rctx for the process, we could take that and aggregate those stats into some type of periodic report. But again, it's not obvious to me what that would look like, and I think it'll be much clearer when we solve the CSRT Logger Matcher expressiveness problem first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236455342


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.

Review Comment:
   The phony metrics are strictly to update the local process's rctx with the appropriate data as a temporary measure until we can chain CSRT contexts into `couch_db_updater`, IOQ, `couch_file`, indexers, etc. I've updated details in the `## Next Steps` README.md



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = couch_srt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = couch_srt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `couch_srt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```
+
+So our `couch_srt_util:tnow/0` is implemented as the following, and we store
+timestamps in `native` format as long as possible to avoid precision loss at
+higher units of time, eg 300 microseconds is zero milliseconds.
+
+```
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+```
+
+We store timestamps in the node's local erlang representation of time,
+specifically to be able to effeciently do time deltas, and then we track time
+deltas from the local node's perspective to not send timestamps across the wire.
+We then utilize `calendar:system_time_to_rfc3339` to convert the local node's
+native time representation to its corresponding time format when we generate the
+process life cycle reports or send an http response.
+
+NOTE: because we do an inline definition and assignment of the
+`#rctx.started_at` and `#rctx.updated_at` fields to `couch_srt_util:tnow()`, we
+_must_ declare `#rctx.updated_at` *after* `#rctx.started_at` to avoid
+fundamental time incongruenties.
+
+### #rctx.started_at = couch_srt_util:tnow() :: integer() | '_',
+
+A static value corresponding to the local node's Erlang monotonic_time at which
+this context was created.
+
+### #rctx.updated_at = couch_srt_util:tnow() :: integer() | '_',
+
+A dynamic value corresponding to the local node's Erlang monotonic_time at which
+this context was updated. Note: unlike `#rctx.started_at`, this value will
+update over time, and in the process lifecycle reports the `#rctx.updated_at`
+value corresponds to the point at which the context was destroyed, allowing for
+calculation of the total duration of the request/context.
+
+### #rctx.pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+
+The primary identifier used to track the resources consumed by a given `pid()`
+for a specific context identified with a `make_ref()`, and combined together as
+unit as a given `pid()`, eg the `chttpd` worker pool, can have many contexts
+over time.
+
+### #rctx.nonce :: nonce() | undefined | '_',
+
+The `Nonce` value of the http request being serviced by the `coordinator_rctx()`
+used as the primary grouping identifier of workers across the cluster, as the
+`Nonce` is funneled through `rexi_server`.
+
+### #rctx.type :: rctx_type() | undefined | '_',
+
+A subtype classifier for the `#rctx{}` contexts, right now only supporting
+`#rpc_worker{}` and `#coordinator{}`, but CSRT was designed to accomodate
+additional context types like `#view_indexer{}`, `#search_indexer{}`,
+`#replicator{}`, `#compactor{}`, `#etc{}`.
+
+### #rctx.dbname :: dbname() | undefined | '_',
+
+The database name, filled in at some point after the initial context creation by
+way of `couch_srt:set_context_dbname/{1,2}`.
+
+### #rctx.username :: username() | undefined | '_',
+
+The requester's username, filled in at some point after the initial context
+creation by way of `couch_srt:set_context_username/{1,2}`.
+
+## Stats Counters
+
+All of these stats counters are stricly `non_neg_integer()` counter values that
+are monotonically increasing, as we only induce positive counter increment calls
+in CSRT. Not all of these values will be nonzero, eg if the context doesn't
+induce Javascript filtering of documents, it won't inc the `#rctx.js_filter`
+field. The `"should_truncate_reports"` config value described in this document
+will conditionally exclude the zero valued fields from being included in the
+process life cycle report.
+
+### #rctx.db_open = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, couch_server, open])
+
+The number of `couch_server:open/2` invocations induced by this context.
+
+### #rctx.docs_read = 0 :: non_neg_integer() | '_',
+
+> Tracking `couch_stats:increment_counter([couchdb, database_reads])
+
+The number of `couch_db:open_doc/3` invocations induced by this context.
+
+### #rctx.docs_written = 0 :: non_neg_integer() | '_',
+
+A phony metric counting docs written by the context, induced by
+`couch_srt:docs_written(length(Docs0)),` in `fabric_rpc:update_docs/3` as a way to
+count the magnitude of docs written, as the actual document writes happen in the
+`#db.main_pid` `couch_db_updater` pid and subprocess tracking is not yet
+supported in CSRT.
+
+This can be replaced with direct counting  once passthrough contexts work.

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236443864


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.

Review Comment:
   Thanks, that's all moved into the dedicated internal docs and cross linked and what not.



##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```

Review Comment:
   Thanks, deleted it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226714622


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").

Review Comment:
   Avoid using macros in the docs, just plain section and k/v strings. Users then can read just one section of the docs and don't have to scroll through to find what the CSRT_MATCHERS_THRESHOLD macro is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226806746


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   Is this the erlang `string()` type or our own locally defined version. I don't remember types very well but looking it up I see `-type string() :: [[char]]` which a list of character so if we emit it as is with jiffy it will produce an array:
   
   ```
   jiffy:encode(#{value => [$a, $b, $c]}).
   <<"{\"value\":[97,98,99]}">>
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236548819


##########
src/couch_srt/src/couch_srt.erl:
##########
@@ -0,0 +1,952 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+%% PidRef API
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+
+%% Context Lifecycle API
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+
+%% Public API
+-export([
+    maybe_track_rexi_init_p/1,
+    maybe_track_local_counter/2,
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+
+%% Stats Collection API
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+
+%% RPC API
+-export([
+    rpc_run/1,
+    rpc_unsafe_run/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-opaque query() :: couch_srt_query:query().
+-opaque query_expression() :: couch_srt_query:query_expression().
+-opaque query_option() :: couch_srt_query:query_option().
+
+%%
+%% RPC Operations
+%%
+
+-spec rpc_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, run, [Query])).
+
+-spec rpc_unsafe_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_unsafe_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, unsafe_run, [Query])).
+
+merge_results(Nodes, Resp) ->
+    %% The result of erpc:multicall is returned as a list where the result from each
+    %% node is placed at the same position as the node name is placed in Nodes.
+    %% That is why we can use `lists:zip/2` here.
+    lists:map(fun format_response/1, lists:zip(Nodes, Resp)).
+
+format_response({Node, {ok, {ok, Result}}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {ok, {error, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {ok, Result}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {error, {erpc, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {Tag, _}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Tag]
+    }.
+
+%%
+%% PidRef Operations
+%%
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    couch_srt_util:get_pid_ref().
+
+-spec get_pid_ref(Rctx :: rctx()) -> pid_ref().
+get_pid_ref(Rctx) ->
+    couch_srt_util:get_pid_ref(Rctx).
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    couch_srt_util:set_pid_ref(PidRef).
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    couch_srt_server:create_pid_ref().
+
+-spec destroy_pid_ref() -> maybe_pid_ref().
+destroy_pid_ref() ->
+    destroy_pid_ref(get_pid_ref()).
+
+%%destroy_pid_ref(undefined) ->
+%%    undefined;
+-spec destroy_pid_ref(PidRef :: maybe_pid_ref()) -> maybe_pid_ref().
+destroy_pid_ref(_PidRef) ->
+    erase(?PID_REF).
+
+%%
+%% Context Lifecycle API
+%%
+
+-spec create_worker_context(From, MFA, Nonce) -> pid_ref() | false when
+    From :: pid_ref(), MFA :: mfa(), Nonce :: nonce().
+create_worker_context(From, {M, F, _A}, Nonce) ->
+    case is_enabled() of
+        true ->
+            Type = #rpc_worker{from = From, mod = M, func = F},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_coordinator_context(Httpd, Path) -> pid_ref() | false when
+    Httpd :: #httpd{}, Path :: list().
+create_coordinator_context(#httpd{method = Verb, nonce = Nonce}, Path0) ->
+    case is_enabled() of
+        true ->
+            Path = list_to_binary([$/ | Path0]),
+            Type = #coordinator{method = Verb, path = Path},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_context(Type :: rctx_type(), Nonce :: term()) -> pid_ref() | false.
+create_context(Type, Nonce) ->
+    Rctx = couch_srt_server:new_context(Type, Nonce),
+    PidRef = get_pid_ref(Rctx),
+    set_pid_ref(PidRef),
+    try
+        couch_srt_util:put_delta_a(Rctx),
+        couch_srt_util:put_updated_at(Rctx),
+        couch_srt_server:create_resource(Rctx),
+        couch_srt_logger:track(Rctx),
+        PidRef
+    catch
+        _:_ ->
+            couch_srt_server:destroy_resource(PidRef),
+            %% calling destroy_context(PidRef) clears the tracker too
+            destroy_context(PidRef),
+            false
+    end.
+
+-spec set_context_dbname(DbName :: binary()) -> boolean().
+set_context_dbname(DbName) ->
+    set_context_dbname(DbName, get_pid_ref()).
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: binary(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_dbname(DbName, PidRef).
+
+-spec set_context_handler_fun(Handler) -> boolean() when
+    Handler :: function() | {atom(), atom()}.
+set_context_handler_fun(Handler) ->
+    set_context_handler_fun(Handler, get_pid_ref()).
+
+-spec set_context_handler_fun(Handler, PidRef) -> boolean() when
+    Handler :: function() | {atom(), atom()}, PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun(Fun, PidRef) when is_function(Fun) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            FProps = erlang:fun_info(Fun),
+            Mod = proplists:get_value(module, FProps),
+            Func = proplists:get_value(name, FProps),
+            set_context_handler_fun({Mod, Func}, PidRef)
+    end;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            couch_srt_server:set_context_handler_fun({Mod, Func}, PidRef)
+    end.
+
+%% @equiv set_context_username(User, get_pid_ref())
+set_context_username(User) ->
+    set_context_username(User, get_pid_ref()).
+
+-spec set_context_username(User, PidRef) -> boolean() when
+    User :: null | undefined | #httpd{} | #user_ctx{} | binary(),
+    PidRef :: maybe_pid_ref().
+set_context_username(null, _) ->
+    false;
+set_context_username(_, undefined) ->
+    false;
+set_context_username(#httpd{user_ctx = Ctx}, PidRef) ->
+    set_context_username(Ctx, PidRef);
+set_context_username(#user_ctx{name = Name}, PidRef) ->
+    set_context_username(Name, PidRef);
+set_context_username(UserName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_username(UserName, PidRef).
+
+-spec destroy_context() -> ok.
+destroy_context() ->
+    destroy_context(get_pid_ref()).
+
+-spec destroy_context(PidRef :: maybe_pid_ref()) -> ok.
+destroy_context(undefined) ->
+    ok;
+destroy_context(PidRef) ->
+    %% Stopping the tracker clears the ets entry for PidRef on its way out
+    couch_srt_logger:stop_tracker(),
+    destroy_pid_ref(PidRef),
+    clear_pdict_markers(),
+    ok.
+
+-spec clear_pdict_markers() -> ok.
+clear_pdict_markers() ->
+    ok = lists:foreach(
+        fun
+            ({{csrt, _} = K, _V}) ->
+                erlang:erase(K);
+            (_) ->
+                ok
+        end,
+        erlang:get()
+    ).
+
+%%
+%% Public API
+%%
+
+-spec maybe_track_rexi_init_p({M, F, A}) -> couch_stats:response() when
+    M :: atom(), F :: atom(), A :: non_neg_integer().
+maybe_track_rexi_init_p({M, F, _A}) ->
+    Metric = [M, F, spawned],
+    case couch_srt:should_track_init_p(Metric) of
+        true -> couch_stats:increment_counter(Metric);
+        false -> ok
+    end.
+
+%% Only potentially track positive increments to counters
+-spec maybe_track_local_counter(any(), any()) -> ok.
+maybe_track_local_counter(Name, Val) when is_integer(Val) andalso Val > 0 ->
+    couch_srt:maybe_inc(Name, Val),
+    ok;
+maybe_track_local_counter(_, _) ->
+    ok.
+
+%% @equiv couch_srt_util:is_enabled().
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    couch_srt_util:is_enabled().
+
+%% @equiv couch_srt_util:is_enabled_reporting().
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    couch_srt_util:is_enabled_reporting().
+
+%% @equiv couch_srt_util:is_enabled_rpc_reporting().
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    couch_srt_util:is_enabled_rpc_reporting().
+
+%% @equiv couch_srt_util:is_enabled_init_p().
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    couch_srt_util:is_enabled_init_p().
+
+-spec get_resource() -> maybe_rctx().
+get_resource() ->
+    get_resource(get_pid_ref()).
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(PidRef) ->
+    couch_srt_server:get_resource(PidRef).
+
+%% Log a CSRT report if any filters match
+-spec maybe_report(ReportName :: string(), PidRef :: pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    couch_srt_logger:maybe_report(ReportName, PidRef).
+
+%% Direct report logic skipping should log filters
+-spec do_report(ReportName :: string(), PidRef :: pid_ref()) -> boolean().
+do_report(ReportName, PidRef) ->
+    couch_srt_logger:do_report(ReportName, get_resource(PidRef)).
+
+-spec to_json(Rctx :: maybe_rctx()) -> map() | null.
+to_json(undefined) ->
+    null;
+to_json(Rctx) ->
+    couch_srt_entry:to_json(Rctx).
+
+%%
+%% Stat Collection API
+%%
+
+-spec inc(Key :: rctx_field()) -> non_neg_integer().
+inc(Key) ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key);
+        false ->
+            0
+    end.
+
+-spec inc(Key :: rctx_field(), N :: non_neg_integer()) -> non_neg_integer().
+inc(Key, N) when is_integer(N) andalso N >= 0 ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key, N);
+        false ->
+            0
+    end.
+
+-spec maybe_inc(Stat :: atom(), Val :: non_neg_integer()) -> non_neg_integer().
+maybe_inc(Stat, Val) ->
+    case maps:is_key(Stat, ?STATS_TO_KEYS) of
+        true ->
+            inc(maps:get(Stat, ?STATS_TO_KEYS), Val);
+        false ->
+            0
+    end.
+
+-spec should_track_init_p(Stat :: [atom()]) -> boolean().
+%% "Example to extend CSRT"
+%% should_track_init_p([fabric_rpc, foo, spawned]) ->
+%%    is_enabled_init_p();
+should_track_init_p([fabric_rpc, all_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, changes, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, get_all_security, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, map_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_doc, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_shard, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, reduce_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, update_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p(_Metric) ->
+    false.
+
+-spec ioq_called() -> non_neg_integer().
+ioq_called() ->
+    inc(ioq_calls).
+
+%% we cannot yet use stats couchdb.query_server.*.ddoc_filter because those
+%% are collected in a dedicated process.
+%% TODO: funnel back stats from background worker processes to the RPC worker
+js_filtered(N) ->
+    inc(js_filter),
+    inc(js_filtered_docs, N).
+
+docs_written(N) ->
+    inc(docs_written, N).
+
+-spec accumulate_delta(Delta :: map() | undefined) -> ok.
+accumulate_delta(Delta) when is_map(Delta) ->
+    is_enabled() andalso couch_srt_server:update_counters(get_pid_ref(), Delta),
+    ok;
+accumulate_delta(undefined) ->
+    ok.
+
+-spec make_delta() -> maybe_delta().
+make_delta() ->
+    case is_enabled() of
+        false ->
+            undefined;
+        true ->
+            couch_srt_util:make_delta(get_pid_ref())
+    end.
+
+-spec rctx_delta(TA :: maybe_rctx(), TB :: maybe_rctx()) -> maybe_delta().
+rctx_delta(TA, TB) ->
+    couch_srt_util:rctx_delta(TA, TB).
+
+%%
+%% Aggregate Query API
+%%
+
+-spec active() -> [rctx()].
+active() ->
+    couch_srt_query:active().
+
+-spec active(Type :: json) -> [rctx()].
+active(Type) ->
+    couch_srt_query:active(Type).
+
+-spec active_coordinators() -> [coordinator_rctx()].
+active_coordinators() ->
+    couch_srt_query:active_coordinators().
+
+%% TODO: cleanup json logic here
+-spec active_coordinators(Type :: json) -> [coordinator_rctx()].
+active_coordinators(Type) ->
+    couch_srt_query:active_coordinators(Type).
+
+-spec active_workers() -> [rpc_worker_rctx()].
+active_workers() ->
+    couch_srt_query:active_workers().
+
+-spec active_workers(Type :: json) -> [rpc_worker_rctx()].
+active_workers(Type) ->
+    couch_srt_query:active_workers(Type).
+
+find_by_nonce(Nonce) ->
+    couch_srt_query:find_by_nonce(Nonce).
+
+find_by_pid(Pid) ->
+    couch_srt_query:find_by_pid(Pid).
+
+find_by_pidref(PidRef) ->
+    couch_srt_query:find_by_pidref(PidRef).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_query:find_workers_by_pidref(PidRef).
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    couch_srt_logger:pid_ref_matchspec(AttrName).
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    couch_srt_logger:pid_ref_attrs(AttrName).
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with couch_srt_logger:pid_ref_attrs/1, and
+%% returning on pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    couch_srt_logger:proc_window(AttrName, Num, Time).
+
+-spec query_matcher(MatcherName :: matcher_name()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName) ->
+    couch_srt_query:query_matcher(MatcherName).
+
+-spec query_matcher(MatcherName :: matcher_name(), Limit :: pos_integer()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName, Limit) ->
+    couch_srt_query:query_matcher(MatcherName, Limit).
+
+%%
+%% Delta API
+%%
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, Delta) ->
+    couch_srt_util:add_delta(T, Delta).
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta(T) ->
+    couch_srt_util:extract_delta(T).
+
+-spec get_delta() -> tagged_delta().
+get_delta() ->
+    couch_srt_util:get_delta(get_pid_ref()).
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    couch_srt_util:maybe_add_delta(T).
+
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, Delta) ->
+    couch_srt_util:maybe_add_delta(T, Delta).
+
+%%
+%% Query API functions
+%%
+%%
+
+%% @doc Construct query from the expressions.
+%% There are following types of expressions allowed in the query.
+%%   <li>group_by/1 @see group_by/1</li>
+%%   <li>group_by/2 @see group_by/1</li>
+%%   <li>sort_by/1 @see sort_by/1</li>
+%%   <li>count_by/1 @see count_by/1</li>
+%%   <li>options/1 @see options/1</li>
+%%   <li>from/1 @see from/1</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec query(QueryExpression :: [query_expression()]) ->
+    query() | {error, any()}.
+query(QueryExpression) ->
+    couch_srt_query:query(QueryExpression).
+
+%% @doc Specify the matcher to use for the query.
+%% If atom 'all' is used then all entries would be in the scope of the query.
+%% Also the use of 'all' makes the query 'unsafe'. Because it scans through all entries
+%% and can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    from("docs_read")
+%% ]),
+%% </code>
+%% @end
+-spec from(MatcherNameOrAll :: string() | all) ->
+    query_expression() | {error, any()}.
+from(MatcherNameOrAll) ->
+    couch_srt_query:from(MatcherNameOrAll).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+group_by(AggregationKeys) ->
+    couch_srt_query:group_by(AggregationKeys).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+group_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:group_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+sort_by(AggregationKeys) ->
+    couch_srt_query:sort_by(AggregationKeys).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+sort_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:sort_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'count_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    count_by(username)
+%% ]),
+%% </code>
+%% @end
+-spec count_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+count_by(AggregationKeys) ->
+    couch_srt_query:count_by(AggregationKeys).
+
+%% @doc Construct 'options' query expression.
+%% There are following types of expressions allowed in the query.
+%%   <li>unlimited/0 @see unlimited/0 (cannot be used with 'with_limit/1')</li>
+%%   <li>with_limit/1 @see with_limit/1 (cannot be used with 'unlimited/0')</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      ...
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec options([query_option()]) ->
+    query_expression() | {error, any()}.
+options(OptionsExpression) ->
+    couch_srt_query:options(OptionsExpression).
+
+%% @doc Enable unlimited number of results from the query.
+%% The use of 'unlimited' makes the query 'unsafe'. Because it can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      unlimited()
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec unlimited() ->
+    query_expression().
+unlimited() ->
+    couch_srt_query:unlimited().
+
+%% @doc Set limit on number of results returned from the query.
+%% The construction of the query fail if the 'limit' is greater than
+%% allowed for this cluster.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      with_limit(100)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec with_limit(Limit :: pos_integer()) ->
+    query_expression() | {error, any()}.
+with_limit(Limit) ->
+    couch_srt_query:with_limit(Limit).
+
+%% @doc Executes provided query. Only 'safe' queries can be executed using 'run'.
+%% The query considered 'unsafe' if any of the conditions bellow are met:
+%%   <li>Query uses 'unlimited/0'</li>
+%%   <li>Query uses 'from(all)'</li>
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% run(Q)
+%% </code>
+%% @end
+-spec run(query()) ->
+    {ok, [{aggregation_key(), pos_integer()}]}
+    | {limit, [{aggregation_key(), pos_integer()}]}.
+run(Query) ->

Review Comment:
   FWIW I think @iilyak did a fantastic job taking my old hacked together HTTP that allows for dynamic aggregations on top of singular fields or groups of fields from the `rctx()` rows, sorted by a different value field. This API is very well tested, has great documentation, excellent type specs, and does a fantastic job on the incoming field validations and properly bubbling up the errors in a manner that is demonstrably well type_spec'ed and handled properly and thoroughly.
   
   Furthermore, he extend that aggregation and filtering logic to be chained on top of using the enabled Logger Matchers to query! This is a fantastic result that allows us to funnel use effecient map specs to first filter the full data set into a targeted reasonable set that can be then aggregated on. The `from("docs_read")` logic handles wrapping the query on top of the effecient ets match spec generated from the configured `docs_read` Logger Matcher, so that the aggregations performed are dependent on our configured cardinality and query limits, _not_ on the number of rctxs being tracked in the ets table.
   
   For example, `run(query(from("ioq_calls"), group_by([username, dbname], ioq_calls), options([with_limit(50)])))))` will load the enabled `ioq_calls` default matcher with the configured `Threshold`, and then funnel that match spec into the ets NIF itself which will then fetch up to `query_cardinality_rows()` from the `ioq_calls` filtered ets tables, and then return the top 50 groups aggregated on `[username, dbname]`, summed on `ioq_calls` and then sorted by top 50.
   
   This is super cool! A significant portion of the logic here is around safely and efficiently translating the provided field names into queryable representations. Again, a fully expressive Mango like syntax that was compiled into a match spec would simplify a lot of these things, but we'd still need a way to do the aggregations and sorting and what not on top of the translation layer. It's not a coincidence that limit is expressed as a simple data structure like `{limit, 1234}`, as that begins to resemble options we can fold over to generate a match spec and make these fully dynamic. More info in wherever you can load http://localhost:8000/csrt/index.html#demonstration-of-expressiveness-constraints-in-logger-matchers-and-ini-settings



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236220507


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.

Review Comment:
   Yeah, I added more details above in https://github.com/apache/couchdb/pull/5602#discussion_r2236175785 but yes, RPC workers track their local stats, ship the deltas across the wire to the coordinator through the rexi calls, and then those RPC workers themselves don't generate a report if the RPC reporting is disable.
   
   The tricky thing is for large `Q` aggregate queries, say `Q=64`, if consistent hashing is uniform, each replica only induces 1/64th of the workload we're matching on the coordinator, which means we have to store a lot more reports to post facto aggregate the data or we do the aggregation up front to get the data to the coordinator process while we're already streaming all the rows.
   
   > Or we just see them at the end of the request
   
   To be very clear: CSRT ships these stats in real time, whenever the (`fabric_rpc`) RPC worker responds through rexi, so that we get the delta updates live as they're happening. This is why it's *CRITICAL* to include the CSRT deltas in the `rexi:ping` function so that long running workers that aren't returning rows will funnel their deltas back to the coordinator during the ping calls so we can maintain a real time update at the coordinator level of the work being induced. This was a core goal given the issues we've seen around long running `_find` queries not returning data until way too late, combined with the client socket timing out and not getting notified until much later. The recent updates to allow for the client to supply the nonce value also means we get that nonce value directly when instantiating the CSRT coordinator context and RPC worker contexts, such that even if the client request disconnects prior to us ever sending the request back through a front load balancer l
 ike HAProxy, we'll still be able to generate the report but importantly we'll generate the report with the provided nonce value, allowing for connections to the original client request that would have otherwise been lostl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236594391


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.

Review Comment:
   Thanks! Then you'll appreciate the even further extend example and additional example I added into the main `csrt/index.rst` documentation page! :D 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236447867


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317652562


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   suggestion: (maybe overkill here, too) gb_trees could be used perhaps as it has `take_largest/1` and `take_smallest/1` functions and automatically keeps the entries sorted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317657477


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.

Review Comment:
   It may be better to be explicit about handling missing ets table and catch error:badarg only



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317584655


##########
src/docs/src/config/csrt.rst:
##########
@@ -0,0 +1,587 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. default-domain:: config
+.. highlight:: ini
+
+.. _config-csrt:
+
+==========================================
+Couch Stats Resource Tracker (CSRT) Config
+==========================================
+
+CSRT configuration options and overview.
+
+.. seealso::
+
+    :doc:`/csrt/index`
+
+CSRT config
+===========
+
+This section contains the top level enablement and configuration options for CSRT.
+
+.. config:section:: csrt :: CSRT Primary Configuration
+
+    .. config:option:: enable :: Enable CSRT data collection and RPC deltas
+
+        Core enablement toggle for CSRT, defaults to false. Enabling this
+        setting initiates local CSRT stats collection as well as shipping deltas
+        in RPC responses to accumulate in the coordinator.
+
+        This does *not* trigger the new RPC spawn metrics, and it does not
+        enable reporting for any of the rctx types.
+
+        .. warning::
+
+            You *MUST* have all nodes in the cluster running a CSRT aware
+            CouchDB *before* you enable it on any node, otherwise the old
+            version nodes won't know how to handle the new RPC formats
+            including an embedded Delta payload.
+
+        Top level CSRT enablement for local data collection and RPC deltas::
+
+            [csrt]
+            enable = false
+
+    .. config:option:: enable_init_p :: Enable RPC spawn metric tracking
+
+        Enablement of tracking new metric counters for different `fabric_rpc` operations
+        spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+        inducing database RPC operations within CouchDB, and these init_p metrics aim to
+        provide node level understandings of the workloads being induced by other
+        coordinator processes. This is especially relevant for databases on subsets of a
+        cluster resulting in non-uniform workloads, these metrics are tailored to
+        provide insight into what work is being spawned on each node in the cluster as a
+        function of time.
+
+        Enablement for tracking counts of spawned RPC workers::
+
+            [csrt]
+            enable_init_p = false
+
+    .. config:option:: enable_reporting :: Enable CSRT Process Lifecyle Reports
+
+        This is the primary toggle for enabling CSRT process lifetime reports
+        containing detailed information about the quantity of work induced by
+        the given request/worker/etc. This is the top level toggle for enabling
+        _any_ reporting, and there also exists
+        :config:option:`csrt/enable_rpc_reporting` to disable the reporting of
+        any individual RPC workers, leaving the coordinator responsible of
+        generating a report with the accumulated deltas.
+
+        .. note::
+
+            Note that this setting toggles whether or not to generate process
+            lifecycle reports, but no reports will be generated until logger
+            matchers have been enabled that trigger a match on CSRT contexts
+            that have surpassed the configured thresholds.
+
+        Top level toggle for whether any process lifecycle reports are generated::
+
+            [csrt]
+            enable_reporting = false
+
+    .. config:option:: enable_rpc_reporting :: Enable RPC process lifecyle reports
+
+        This enables the possibility of RPC workers generating reports. They
+        still need to hit the configured thresholds to induce a report, but
+        this will generate CSRT process lifetime reports for individual RPC
+        workers that trigger the configured logger thresholds. This allows for
+        quantifying per node resource usage when desired, as otherwise the
+        reports are at the http request level and don't provide per node stats.
+
+        The key idea here is that having RPC level CSRT process lifetime
+        reporting is incredibly useful, but can also generate large quantities
+        of data. For example, a view query on a Q=64 database will stream
+        results from 64 shard replicas, resulting in at least 64 RPC reports,
+        plus any that might have been generated from RPC workers that "lost"
+        the race for shard replica. This is very useful, but a lot of data
+        given the verbose nature of funneling it through the RSyslog reports,
+        however, the ability to write directly to something like ClickHouse or
+        another columnar store would be great.
+
+        Until there's an efficient storage mechanism to stream the results to,
+        the rsyslog entries work great and are very practical, but care must be
+        taken to not generate too much data for aggregate queries as they
+        generate at least `Qx` more report than an individual report per http
+        request from the coordinator.  This setting exists as a way to either
+        a) utilize the logger matcher configured thresholds to allow for _any_
+        rctx's to be recorded when they induce heavy operations, either
+        Coordinator or RPC worker; or b) to _only_ log workloads at the
+        coordinator level.
+
+        .. note::
+
+            This setting exists because we lack an expressive enough config
+            declaration to easily chain the matchspec constructions as
+            `ets:fun2ms/1` is a special compile time parse transform macro that
+            requires the full definition to be specified directly, it cannot
+            be interactively constructed. That said, you _can_ register matchers
+            through `remsh` with more specific and fine grained pattern matching,
+            and a more expressive system for defining matchers are being
+            explored.
+
+        .. warning::
+
+            Enabling this setting *will* generate considerably more logs! Specifically, for aggregate queries and database operations, this will generate `Q` * `N` times more logs than a singular doc request taking only `N` inreacting with a singular shard range. See the note above about this being a temporary setting during the experimental stages of CSRT.
+
+        Toggle to enable possibility of RPC process lifecycle reports::
+
+            [csrt]
+            enable_rpc_reporting = false
+
+    .. config:option:: should_truncate_reports :: truncate zero values from lifecyle reports
+
+        enables truncation of the csrt process lifetime reports to not include
+        any fields that are zero at the end of process lifetime, eg don't
+        include `js_filter=0` in the report if the request did not induce
+        javascript filtering.
+
+        this can be disabled if you really care about consistent fields in the
+        report logs, but this is a log space saving mechanism, similar to
+        disabling rpc reporting by default, as its a simple way to reduce
+        overall volume
+
+        Truncate zero values from process lifecycle reports, enabled by default:
+
+            [csrt]
+            should_truncate_reports = true
+
+    .. config:option:: query_limit :: Maximum quantity of rows to return in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_limit = 100
+
+    .. config:option:: query_cardinality_limit :: Maximum quantity of rows to allow in CSRT query/http requests.
+
+        Limit the quantity of rows that can be loaded in an http query.::
+
+            [csrt]
+            query_cardinality_limit = 10000
+
+.. _csrt-logger-matcher-configuration:
+
+CSRT Logger Matcher Configuration
+=================================
+
+There are currently eight builtin default logger matchers designed to make it
+easy to do filtering on heavy resource usage inducing and long running
+requests. These are designed as a simple baseline of useful matchers, declared
+in a manner amenable to `default.ini` based constructs. More expressive matcher
+declarations are being explored, and matchers of arbitrary complexity can be
+registered directly through `remsh`. The default matchers are all designed around
+an integer config Threshold that triggers on a specific field, eg docs read, or
+on a delta of fields for long requests and changes requests that process many
+rows but return few.
+
+The current default matchers are:
+
+  * `all_coordinators`: match all Coordinators handling HTTP requests
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/all_coordinators>` | none
+
+  * `all_rpc_workers`: match all RPC Worker handling internal requests
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/all_rpc_workers>` | none
+
+  * `docs_read`: match all requests reading more than N docs
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/docs_read>` | :config:option:`Threshold <csrt_logger.matchers_threshold/docs_read>`
+
+  * `rows_read`: match all requests reading more than N rows
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/rows_read>` | :config:option:`Threshold <csrt_logger.matchers_threshold/rows_read>`
+
+  * `docs_written`: match all requests writing more than N docs
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/docs_written>` | :config:option:`Threshold <csrt_logger.matchers_threshold/docs_written>`
+
+  * `ioq_calls`: match all requests inducing more than N ioq_calls
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/ioq_calls>` | :config:option:`Threshold <csrt_logger.matchers_threshold/ioq_calls>`
+
+  * `long_reqs`: match all requests lasting more than N milliseconds
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/long_reqs>` | :config:option:`Threshold <csrt_logger.matchers_threshold/long_reqs>`
+
+  * `changes_processed`: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+
+    * :config:option:`Enable <csrt_logger.matchers_enabled/changes_processed>` | :config:option:`Threshold <csrt_logger.matchers_threshold/changes_processed>`
+
+Each of the default matchers has an enablement setting in
+ref:`csrt-logger-matcher-configuration-enablement` for toggling enablement of

Review Comment:
   May need another `:` in front as it doesn't render as a link reference currently.
   
   <img width="574" height="106" alt="Screenshot 2025-09-02 at 10 11 07 PM" src="https://github.com/user-attachments/assets/7deaeee1-2dd3-42a3-b972-248f4c4f3507" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317623602


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,840 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occur in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in retroactively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a database `foo` with 11k documents containing a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy processes, live, as well as a process lifecycle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecycle reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_srt/src/couch_srt_query.erl#L412-L740
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxiliary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persistent_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works efficiently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+These tools provide a direct and conditional mechanism to generate a report,
+with `do_report/2`, and `maybe_report`, respectively, with the latter testing
+the provided `rctx()` against the actively registered Logger Matchers.
+
+The `is_enabled*` checks perform the various enablement checks, as described in
+the corresponding Config documentation sections for the fields.
+
+And lastly, is the `couch_srt:to_json/1` function which takes a `maybe_rctx()` as
+opposed to `couch_srt_entry:to_json/1` which only takes an actual `rctx()`,
+specifically to make it easier to map `couch_srt:to_json/1` to output from
+`couch_srt:proc_window/3` and easily handle the case when
+`couch_srt:get_resource/1` returns undefined in the event the context has
+already exited before we could look at it.
+
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and efficient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an efficient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+especially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as `started_at` and `updated_at` timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing

Review Comment:
   tiny nit: not sure if we wanted to capitalize `Stats Counters`? Is that a variable may it could be encased in backticks to have a monospaced font?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316639213


##########
src/couch_srt/src/couch_srt_logger.erl:
##########
@@ -0,0 +1,584 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_logger).
+
+-behaviour(gen_server).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            couch_srt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}, infinity).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case couch_srt_util:is_enabled() andalso couch_srt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->

Review Comment:
   > find_matches(Rctxs, #{}=Matchers) when is_list(Rctxs) ->
   > which now has conflicted uses of guard clauses
   
   Hmm, I don't see how they are conflicting? One is a map and the other is a list and it's immediately obvious. It just that in the rest of the code-base we use the `#{}` mostly. But it's fine, it's minor nit if you think is_map looks better let's keep it.
   
   



##########
src/chttpd/src/chttpd_httpd_handlers.erl:
##########
@@ -20,6 +20,7 @@ url_handler(<<"_utils">>) -> fun chttpd_misc:handle_utils_dir_req/1;
 url_handler(<<"_all_dbs">>) -> fun chttpd_misc:handle_all_dbs_req/1;
 url_handler(<<"_dbs_info">>) -> fun chttpd_misc:handle_dbs_info_req/1;
 url_handler(<<"_active_tasks">>) -> fun chttpd_misc:handle_task_status_req/1;
+url_handler(<<"_active_resources">>) -> fun couch_srt_httpd:handle_resource_status_req/1;

Review Comment:
   Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221196208


##########
src/couch_stats/src/csrt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case csrt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling csrt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case csrt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = csrt_util:tnow(),
+            csrt_util:put_updated_at(Now),
+            UpdatedInc = csrt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(_, _From, State) ->
+    {reply, ok, State}.

Review Comment:
   I used to hold that belief but Erlang OTP made most of these callbacks optional with the default behavior to silently ignore everything, so that's the behavior I run with these days. Eg https://www.erlang.org/doc/apps/stdlib/gen_server.html#c:handle_info/2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223230675


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]
+; For example:
+; foo = 100
+; _dbs = 123
+; _users = 234
+; foo/bar = 200
+
+[csrt_logger.matchers_enabled]
+;docs_read = false
+;rows_read = false
+;docs_written = false
+;long_reqs = false
+;changes_processed = false
+;ioq_calls = false
+
+[csrt_logger.matchers_threshold]

Review Comment:
   Maybe add a short comment next to the section about the difference between the two.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223224404


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]

Review Comment:
   I think it's a bit confusing have both there so maybe add a comment about these being either if using rpc worker matcher or cluster db name if using coordinator matches.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2223623754


##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format
+-spec tutc(Time :: integer()) -> calendar:rfc3339_string().
+tutc(Time0) when is_integer(Time0) ->
+    Unit = millisecond,
+    Time1 = Time0 + erlang:time_offset(),
+    Time = erlang:convert_time_unit(Time1, native, Unit),
+    calendar:system_time_to_rfc3339(Time, [{unit, Unit}, {offset, "z"}]).
+
+%% Returns dt (delta time) in microseconds
+%% @equiv make_dt(A, B, microsecond)
+-spec make_dt(A, B) -> pos_integer() when
+    A :: integer(),
+    B :: integer().
+make_dt(A, B) ->
+    make_dt(A, B, microsecond).
+
+%% Returns monotonic dt (delta time) in specified time_unit()
+-spec make_dt(A, B, Unit) -> pos_integer() when
+    A :: integer(),
+    B :: integer(),
+    Unit :: erlang:time_unit().
+make_dt(A, A, _Unit) when is_integer(A) ->
+    %% Handle edge case when monotonic_time()'s are equal
+    %% Always return a non zero value so we don't divide by zero
+    %% This always returns 1, independent of unit, as that's the smallest
+    %% possible positive integer value delta.
+    1;
+make_dt(A, B, Unit) when is_integer(A) andalso is_integer(B) andalso B > A ->
+    case erlang:convert_time_unit(abs(B - A), native, Unit) of
+        Delta when Delta > 0 ->
+            Delta;
+        _ ->
+            %% Handle case where Delta is smaller than a whole Unit, eg:
+            %% Unit = millisecond,
+            %% (node1@127.0.0.1)2> erlang:convert_time_unit(423, native, Unit).
+            %% 0
+            1
+    end.
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, undefined) ->
+    T;
+add_delta(T, Delta) when is_map(Delta) ->
+    add_delta_int(T, {delta, Delta}).
+
+-spec add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+add_delta_int(T, {delta, _} = Delta) ->
+    {T, Delta}.
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta({Msg, {delta, Delta}}) ->
+    {Msg, Delta};
+extract_delta(Msg) ->
+    {Msg, undefined}.
+
+-spec get_delta(PidRef :: maybe_pid_ref()) -> tagged_delta().
+get_delta(PidRef) ->
+    {delta, make_delta(PidRef)}.
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            maybe_add_delta_int(T, get_delta(get_pid_ref()))
+    end.
+
+%% Allow for externally provided Delta in error handling scenarios
+%% eg in cases like rexi_server:notify_caller/3
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, undefined) ->
+    T;
+maybe_add_delta(T, Delta0) when is_map(Delta0) ->
+    case is_enabled() of
+        false ->
+            T;
+        true ->
+            Delta = {delta, Delta0},
+            maybe_add_delta_int(T, Delta)
+    end.
+
+-spec maybe_add_delta_int(T :: term(), Delta :: tagged_delta()) -> term_delta().
+maybe_add_delta_int(T, {delta, undefined}) ->
+    T;
+maybe_add_delta_int(T, {delta, _} = Delta) ->
+    add_delta_int(T, Delta).
+
+-spec make_delta(PidRef :: maybe_pid_ref()) -> maybe_delta().
+make_delta(undefined) ->
+    undefined;
+make_delta(PidRef) ->
+    TA = get_delta_a(),
+    TB = csrt_server:get_resource(PidRef),
+    Delta = rctx_delta(TA, TB),
+    put_delta_a(TB),
+    Delta.
+
+-spec rctx_delta(TA :: Rctx, TB :: Rctx) -> map().
+rctx_delta(#rctx{} = TA, #rctx{} = TB) ->
+    Delta = #{
+        docs_read => TB#rctx.docs_read - TA#rctx.docs_read,
+        docs_written => TB#rctx.docs_written - TA#rctx.docs_written,
+        js_filter => TB#rctx.js_filter - TA#rctx.js_filter,
+        js_filtered_docs => TB#rctx.js_filtered_docs - TA#rctx.js_filtered_docs,
+        rows_read => TB#rctx.rows_read - TA#rctx.rows_read,
+        changes_returned => TB#rctx.changes_returned - TA#rctx.changes_returned,
+        get_kp_node => TB#rctx.get_kp_node - TA#rctx.get_kp_node,
+        get_kv_node => TB#rctx.get_kv_node - TA#rctx.get_kv_node,
+        db_open => TB#rctx.db_open - TA#rctx.db_open,
+        ioq_calls => TB#rctx.ioq_calls - TA#rctx.ioq_calls,
+        %% "Example to extend CSRT"
+        %% write_kp_node => TB#rctx.write_kp_node - TA#rctx.write_kp_node,
+        %% write_kv_node => TB#rctx.write_kv_node - TA#rctx.write_kv_node,
+        dt => make_dt(TA#rctx.updated_at, TB#rctx.updated_at)
+    },
+    %% TODO: reevaluate this decision
+    %% Only return non zero (and also positive) delta fields
+    %% NOTE: this can result in Delta's of the form #{dt => 1}
+    maps:filter(fun(_K, V) -> V > 0 end, Delta);
+rctx_delta(_, _) ->
+    undefined.
+
+-spec get_delta_a() -> maybe_rctx().
+get_delta_a() ->
+    erlang:get(?DELTA_TA).
+
+-spec put_delta_a(TA :: rctx()) -> maybe_rctx().
+put_delta_a(TA) ->
+    erlang:put(?DELTA_TA, TA).
+
+-spec get_updated_at() -> maybe_integer().
+get_updated_at() ->
+    erlang:get(?LAST_UPDATED).
+
+-spec put_updated_at(Updated :: rctx() | integer()) -> maybe_integer().
+put_updated_at(#rctx{updated_at = Updated}) ->
+    put_updated_at(Updated);
+put_updated_at(Updated) when is_integer(Updated) ->
+    erlang:put(?LAST_UPDATED, Updated).
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    get(?PID_REF).
+
+-spec get_pid_ref(Rctx :: rctx()) -> maybe_pid_ref().
+get_pid_ref(#rctx{pid_ref = PidRef}) ->
+    PidRef;
+get_pid_ref(_) ->
+    undefined.
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    erlang:put(?PID_REF, PidRef),
+    PidRef.
+
+%% @equiv set_fabric_init_p(Func, Enabled, true).
+-spec set_fabric_init_p(Func :: atom(), Enabled :: boolean()) -> ok.
+set_fabric_init_p(Func, Enabled) ->
+    set_fabric_init_p(Func, Enabled, true).
+
+%% Expose Persist for use in test cases outside this module
+-spec set_fabric_init_p(Func, Enabled, Persist) -> ok when
+    Func :: atom(), Enabled :: boolean(), Persist :: boolean().
+set_fabric_init_p(Func, Enabled, Persist) ->
+    Key = fabric_conf_key(Func),
+    ok = config:set_boolean(?CSRT_INIT_P, Key, Enabled, Persist).
+
+-spec fabric_conf_key(Key :: atom()) -> string().
+fabric_conf_key(Key) ->
+    %% Double underscore to separate Mod and Func
+    "fabric_rpc__" ++ atom_to_list(Key).

Review Comment:
   Simpler is better but for what it's worth the ":" should work as well. I verified it both in config:get() remsh, and HTTP PUT/GET context
   
   ```
   % curl -XPUT -H'Content-type:application/json' $DB/_node/_local/_config/bogus/m2:f2 -d'"200"'
   
   {
       "m2:f2": "200",
       "m:f": "100"
   }
   ```
   
   ```
   http $DB/_node/_local/_config/bogus
   
   {
       "m:f": "100"
   }
   ```
   
   
   ```
   > config:set("bogus", "m:f", "100").
   ```
   
   ```
   > config:get("bogus").
   [{"m:f","100"}]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220898109


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([

Review Comment:
   I've updated this in debe52fab as well as linked out to the excellent csrt_query docs and Eunit tests you wrote.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220958232


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true

Review Comment:
   These particular values are enabled for CI purposes during this PR review, the correct default values are immediately below these and will be deleted prior to merging. See my top level PR response to your comments, I've included a local diff I have ready to flip this, but again, we need this to get a green CI with CSRT enabled.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221072289


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.

Review Comment:
   Yes, the key idea with CSRT is that it's a real time stats tracking engine where all the callers that induce `couch_stats:increment_counter` calls also immediately induce the corresponding CSRT call for tracking at the process level. This makes those values immediately available. The final "report" is just reading from those values once the process is dead or the context is closed, and then writing them to a report.
   
   This is especially critical for `rexi:ping`, as we _need_ to be streaming the induced deltas over the wire through `rexi:ping` to keep the coordinators apprised of the usage of long running requests. This was specifically done so that we still get real time updates for long running `_find` queries that rarely return data. Checkout the comments in `rexi:ping`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221219324


##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).

Review Comment:
   Good call, fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221247786


##########
src/couch_stats/src/csrt_util.erl:
##########
@@ -0,0 +1,386 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_util).
+
+-export([
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1,
+    should_track_init_p/2,
+    tnow/0,
+    tutc/0,
+    tutc/1
+]).
+
+%% Delta API
+-export([
+    add_delta/2,
+    extract_delta/1,
+    get_delta/1,
+    get_delta_a/0,
+    get_updated_at/0,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    make_delta/1,
+    make_dt/2,
+    make_dt/3,
+    rctx_delta/2,
+    put_delta_a/1,
+    put_updated_at/1
+]).
+
+%% Extra niceties and testing facilities
+-export([
+    set_fabric_init_p/2,
+    set_fabric_init_p/3
+]).
+
+-include_lib("csrt.hrl").
+
+-ifdef(TEST).
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% randomly enable CSRT during testing to handle unexpected failures
+    case config:get_boolean(?CSRT, "randomize_testing", true) of
+        true ->
+            rand:uniform(100) > 80;
+        false ->
+            config:get_boolean(?CSRT, "enable", true)
+    end.
+-else.
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable", true).
+-endif.
+
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_init_p", true).
+
+-spec should_track_init_p(Mod :: atom(), Func :: atom()) -> boolean().
+should_track_init_p(fabric_rpc, Func) ->
+    is_enabled_init_p() andalso config:get_boolean(?CSRT_INIT_P, fabric_conf_key(Func), false);
+should_track_init_p(_Mod, _Func) ->
+    false.
+
+%% Toggle to disable all reporting
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    %% TODO: toggle back to false before merging
+    config:get_boolean(?CSRT, "enable_reporting", true).
+
+%% Toggle to disable all reporting from #rpc_worker{} types, eg only log
+%% #coordinator{} types. This is a bit of a kludge that would be better served
+%% by a dynamic match spec generator, but this provides a know for disabling
+%% any rpc worker logs, even if they hit the normal logging Threshold's.
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    config:get_boolean(?CSRT, "enable_rpc_reporting", false).
+
+%% Monotnonic time now in native format using time forward only event tracking
+-spec tnow() -> integer().
+tnow() ->
+    erlang:monotonic_time().
+
+%% Get current system time in UTC RFC 3339 format
+-spec tutc() -> calendar:rfc3339_string().
+tutc() ->
+    tutc(tnow()).
+
+%% Convert a integer system time in milliseconds into UTC RFC 3339 format

Review Comment:
   Aha, good catch and suggestion! I've fixed that and updated the comment to drop mention of milliseconds. That was from when the deltas were in milliseconds rather than native format, I switched that over a while back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2217007054


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([

Review Comment:
   I think this list is outdated.
   
   ```erlang
   % Aggregate Query API
   -export([
       active/0,
       active/1,
       active_coordinators/0,
       active_coordinators/1,
       active_workers/0,
       active_workers/1,
       find_by_nonce/1,
       find_by_pid/1,
       find_by_pidref/1,
       find_workers_by_pidref/1,
       query_matcher/1,
       query_matcher/2
   ]).
   
   -export([
       query/1,
       from/1,
       group_by/1,
       group_by/2,
       sort_by/1,
       sort_by/2,
       count_by/1,
       options/1,
       unlimited/0,
       with_limit/1,
   ]).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317623602


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,840 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occur in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in retroactively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a database `foo` with 11k documents containing a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy processes, live, as well as a process lifecycle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecycle reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_srt/src/couch_srt_query.erl#L412-L740
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxiliary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/da87fc3fd7beb79f1ba63cf430dd92818fb02a62/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persistent_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works efficiently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`couch_srt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`couch_srt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+These tools provide a direct and conditional mechanism to generate a report,
+with `do_report/2`, and `maybe_report`, respectively, with the latter testing
+the provided `rctx()` against the actively registered Logger Matchers.
+
+The `is_enabled*` checks perform the various enablement checks, as described in
+the corresponding Config documentation sections for the fields.
+
+And lastly, is the `couch_srt:to_json/1` function which takes a `maybe_rctx()` as
+opposed to `couch_srt_entry:to_json/1` which only takes an actual `rctx()`,
+specifically to make it easier to map `couch_srt:to_json/1` to output from
+`couch_srt:proc_window/3` and easily handle the case when
+`couch_srt:get_resource/1` returns undefined in the event the context has
+already exited before we could look at it.
+
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `couch_srt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## Query API
+
+See the `Additional Overview and Examples` section above for more details.
+
+```
+% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+```
+
+## couch_srt:proc_window/3 -- Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `couch_srt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and efficient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% couch_srt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an efficient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+especially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`couch_srt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as `started_at` and `updated_at` timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing

Review Comment:
   tiny nit: not sure if we wanted to capitalize `Stats Counters`? Is that a variable may it could be encased in backticks to have a monospaced font?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2323444180


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.

Review Comment:
   Yes, the catch is to protect against missing table. If table is missing we don't care about the events we would miss. The main goal is to keep node stable, crashing here is not acceptable. However ets might throw other exceptions as well:
   
   - system_limit
   - badarg
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2316517635


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `couch_srt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.

Review Comment:
   Very nice, that makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2317656301


##########
src/couch_srt/src/couch_srt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case couch_srt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.

Review Comment:
   The catch is probably intended to protect against an undefined ets table, wonder if it's better to be explicit about and catch a badarg error?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2226699616


##########
src/couch_srt/README.md:
##########
@@ -0,0 +1,1025 @@
+# couch_srt: Couch Stats Resource Tracker aka CSRT
+
+The `couch_srt` app introduces the Couch Stats Resource Tracker, aka CSRT for
+short. CSRT is a real time stats tracking system that tracks the quantity of
+resources induced at the process level in a live queryable manner, while also
+generating process lifetime reports containing statistics on the total resource
+load of a request, as a function of CouchDB operations like dbs/docs opened,
+view and changes rows read, changes returned vs processed, Javascript filter
+usage, request duration, and more. This system is a paradigm shift in CouchDB
+visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `couch_srt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+## A Simple Example
+
+Given a databse `foo` with 11k documents containg a `doc.value` field that is an
+integer value which can be filtered in a design doc by way of even and odd. If
+we instantiate a series of while loops in parallel making requests of the form:
+
+> GET /foo/_changes?filter=bar/even&include_docs=true
+
+We can generate a good chunk of load on a local laptop dev setup, resulting in
+requests that take a few seconds to load through the changes feed, fetch all 11k
+docs, and then funnel them through the Javascript engine to filter for even
+valued docs; this allows us time to query these heavier requests live and see
+them in progress with the real time stats tracking and querying capabilities of
+CSRT.
+
+For example, let's use `couch_srt:proc_window/3` as one would do with
+`recon:proc_window/3` to get an idea of the heavy active processes on the
+system:
+
+```
+(node1@127.0.0.1)2> rp([{PR, couch_srt:to_json(couch_srt:get_resource(PR))} || {PR, _, _} <- couch_srt:proc_window(ioq_calls, 3, 1000)]).
+[{{<0.5090.0>,#Ref<0.2277656623.605290499.37969>},
+  #{changes_returned => 3962,db_open => 10,dbname => <<"foo">>,
+    docs_read => 7917,docs_written => 0,get_kp_node => 54,
+    get_kv_node => 1241,ioq_calls => 15834,js_filter => 7917,
+    js_filtered_docs => 7917,nonce => <<"cc5a814ceb">>,
+    pid_ref =>
+        <<"<0.5090.0>:#Ref<0.2277656623.605290499.37969>">>,
+    rows_read => 7917,
+    started_at => <<"2025-07-21T17:25:08.784z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5087.0>,#Ref<0.2277656623.606601217.92191>},
+  #{changes_returned => 4310,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8624,docs_written => 0,get_kp_node => 58,
+    get_kv_node => 1358,ioq_calls => 17248,js_filter => 8624,
+    js_filtered_docs => 8624,nonce => <<"0e625c723a">>,
+    pid_ref =>
+        <<"<0.5087.0>:#Ref<0.2277656623.606601217.92191>">>,
+    rows_read => 8624,
+    started_at => <<"2025-07-21T17:25:08.424z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}},
+ {{<0.5086.0>,#Ref<0.2277656623.605290499.27728>},
+  #{changes_returned => 4285,db_open => 10,dbname => <<"foo">>,
+    docs_read => 8569,docs_written => 0,get_kp_node => 57,
+    get_kv_node => 1349,ioq_calls => 17138,js_filter => 8569,
+    js_filtered_docs => 8569,nonce => <<"962cda1645">>,
+    pid_ref =>
+        <<"<0.5086.0>:#Ref<0.2277656623.605290499.27728>">>,
+    rows_read => 8569,
+    started_at => <<"2025-07-21T17:25:08.406z">>,
+    type =>
+        <<"coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes">>,
+    updated_at => <<"2025-07-21T17:25:13.051z">>,
+    username => <<"adm">>}}]
+ok
+```
+
+This shows us the top 3 most active processes (being tracked in CSRT) over the
+next 1000 milliseconds, sorted by number of `ioq_calls` induced! All of three
+of these processes are incurring heavy usage, reading many thousands of docs
+with 15k+ IOQ calls and heavy JS filter usage, exactly the types of requests
+you want to be alerted to. CSRT's proc window logic is built on top of Recon's,
+which doesn't return the process info itself, so you'll need to fetch the
+process status with `couch_srt:get_resource/1` and then pretty print it with
+`couch_srt:to_json/1`.
+
+The output above is a real time snapshot of the live running system and shows
+processes actively inducing additional resource usage, so these CSRT context
+values are just a time snapshot of where that process was at, as of the
+`updated_at` timestamp. We can reference the nonce value to search through the
+report logs for a final report, assuming the given context ended up using
+sufficient resources to trigger a logger matcher lifetime report. The above
+changes requests were induced specifically to induce reports as well, so
+unsurprisingly we have reports for all three.
+
+However, I want to first show the existing visibility into these changes
+requests exposed by the raw HTTP logs to highlight the impact of the CSRT
+reports and new visibility into request workloads exposed.
+
+First, let's look at the existing HTTP logs for those 3 requests:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9872)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep -v '^\[report]'
+[notice] 2025-07-21T17:25:14.520641Z node1@127.0.0.1 <0.5087.0> 0e625c723a localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6096
+[notice] 2025-07-21T17:25:14.521417Z node1@127.0.0.1 <0.5086.0> 962cda1645 localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6115
+[notice] 2025-07-21T17:25:14.844317Z node1@127.0.0.1 <0.5090.0> cc5a814ceb localhost:15984 127.0.0.1 adm GET /foo/_changes?filter=bar/even&asdf=fdsa&include_docs=true 200 ok 6059
+```
+
+So we see the requests were made, and we can see it's doing `include_docs=true`
+as well as using a customer filter, both obvious indications that this is a
+potentially heavier request, however, we don't know if database foo had a
+thousand docs or a billion docs, whether those docs were small or large, nor any
+indication of the computational complexity of the reference filter function.
+This makes it challenging to retroactively correlate heavy resource usage at a
+hardware level with the underlying requests that induced those workloads,
+especially if the heavy requests are an inconspicuous subset of the full
+database workload.
+
+CSRT resolves this by providing a real time querying system to find the active
+heavy proceses, live, as well as a process lifecyle reporting engine providing
+detailed analysis of the workloads induced by the request.
+
+Let's assume we had the default IOQ logger matcher enabled, with the default
+configuration of logging any requests inducing more than 10k IOQ calls, which
+would catch all three of our requests above, even though they're all still
+going. As a result, we generate process lifecylce reports for all three of those
+requests, as we can see:
+
+```
+(chewbranca)-(jobs:1)-(~/src/couchdb_csrt_v3)
+(! 9873)-> grep 'cc5a814ceb\|0e625c723a\|962cda1645' ./dev/logs/node1.log | grep '^\[report]'
+[report] 2025-07-21T17:25:14.520787Z node1@127.0.0.1 <0.5174.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="0e625c723a" pid_ref="<0.5087.0>:#Ref<0.2277656623.606601217.92191>" rows_read=11001 started_at="2025-07-21T17:25:08.424z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.520z" username="adm"]
+[report] 2025-07-21T17:25:14.521578Z node1@127.0.0.1 <0.5155.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="962cda1645" pid_ref="<0.5086.0>:#Ref<0.2277656623.605290499.27728>" rows_read=11001 started_at="2025-07-21T17:25:08.406z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.521z" username="adm"]
+[report] 2025-07-21T17:25:14.844436Z node1@127.0.0.1 <0.5213.0> -------- [csrt-pid-usage-lifetime changes_returned=5500 db_open=10 dbname="foo" docs_read=11001 get_kp_node=72 get_kv_node=1754 ioq_calls=22002 js_filter=11001 js_filtered_docs=11001 nonce="cc5a814ceb" pid_ref="<0.5090.0>:#Ref<0.2277656623.605290499.37969>" rows_read=11001 started_at="2025-07-21T17:25:08.784z" type="coordinator-{chttpd_db:handle_changes_req}:GET:/foo/_changes" updated_at="2025-07-21T17:25:14.844z" username="adm"]
+```
+
+We find the process lifecycle reports for the requests with the three grep'ed on
+nonces, and we can see they all read the 11k core documents, plus the one design
+document, JS filtered all 11,001 docs, and then only returned the 5500 doc's
+containing an even `doc.value` field.
+
+This also shows the discrepancy between the quantity of induced resource usage
+to actually generate a request, relative to the magnitude of the data returned.
+All of our `doc.value` fields were positive integers, if we had a filter
+function searching for negative `doc.value` results, we would have found none,
+resulting in `changes_returned=0`, but we would have still induced the 11,001
+doc loads and Javascript filter calls.
+
+CSRT is specifically built to automatically find and report these types of
+workload discrepancies and in general to help highlight where individual HTTP
+requests use drastically more resources than the median workloads.
+
+See the dedicated proc window documentation section further down for more info.
+
+## Additional Overview and Examples
+
+The query and HTTP API's are well documented and tested (h/t @iilyak) and
+provide an excellent overview of the interaction patterns and query capabilities
+of CSRT. Those can be found at:
+
+* `couch_srt_query.erl` "Query API functions"
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/src/couch_srt_query.erl#L319-L674
+  - the above highlighted functions are well tested, typespec'ed, and have
+    auxilary documentation and examples, an excellent resource
+* the `couch_srt_query_tests.erl` Eunit tests are an excellent overview of utilizing
+  the `couch_srt_query:` API from Erlang to find, filter, and aggregate CSRT real
+  time contexts
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_query_tests.erl
+* similarly, the `couch_srt_httpd_tests.erl` Eunit tests are an excellent overview of
+  performing the same style `couch_srt_query:` queries, but through the HTTP API
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_httpd_tests.erl
+* Additionally there's the `couch_srt_logger_tests.erl` Eunit tests which demonstrate
+  the different default logger matchers in action
+  - https://github.com/apache/couchdb/blob/93bc894380056ccca1f77415454e991c4d914249/src/couch_stats/test/eunit/couch_srt_logger_tests.erl
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbnames_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reports at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `couch_srt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+spawned by way of `rexi_server:init_p/3`. This is the primary mechanism for
+inducing database RPC operations within CouchDB, and these init_p metrics aim to
+provide node lever understandings of the workloads being induced by other
+coordinator proceses. This is especially relevant for databases on subsets of a
+cluster resulting in non-uniform workloads, these metrics are tailored to
+provide insight into what work is being spawned on each node in the cluster as a
+function of time.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.

Review Comment:
   That's why I was thinking maybe having a per/node per/time window summary mode might help to avoid the logs being overloaded and still get the rpc-level data. Kind of a middle ground or if that works out it could be just the default (one mode) so we (users) won't have to choose between only vs coordinator + enable_rpc_reporting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236458686


##########
src/couch_srt/src/couch_srt.erl:
##########
@@ -0,0 +1,952 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt).
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+%% PidRef API
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+
+%% Context Lifecycle API
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+
+%% Public API
+-export([
+    maybe_track_rexi_init_p/1,
+    maybe_track_local_counter/2,
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    is_enabled_reporting/0,
+    is_enabled_rpc_reporting/0,
+    maybe_report/2,
+    to_json/1
+]).
+
+%% Stats Collection API
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+
+%% RPC API
+-export([
+    rpc_run/1,
+    rpc_unsafe_run/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    query_matcher/1,
+    query_matcher/2
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-export([
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-opaque query() :: couch_srt_query:query().
+-opaque query_expression() :: couch_srt_query:query_expression().
+-opaque query_option() :: couch_srt_query:query_option().
+
+%%
+%% RPC Operations
+%%
+
+-spec rpc_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, run, [Query])).
+
+-spec rpc_unsafe_run(Query :: query()) ->
+    [
+        #{
+            node => node(),
+            result => [{aggregation_key(), pos_integer()}],
+            errors => [atom()]
+        }
+    ].
+rpc_unsafe_run(Query) ->
+    Nodes = mem3:nodes(),
+    merge_results(Nodes, erpc:multicall(Nodes, ?MODULE, unsafe_run, [Query])).
+
+merge_results(Nodes, Resp) ->
+    %% The result of erpc:multicall is returned as a list where the result from each
+    %% node is placed at the same position as the node name is placed in Nodes.
+    %% That is why we can use `lists:zip/2` here.
+    lists:map(fun format_response/1, lists:zip(Nodes, Resp)).
+
+format_response({Node, {ok, {ok, Result}}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {ok, {error, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {ok, Result}}) ->
+    #{
+        node => Node,
+        result => Result,
+        errors => []
+    };
+format_response({Node, {error, {erpc, Reason}}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Reason]
+    };
+format_response({Node, {Tag, _}}) ->
+    #{
+        node => Node,
+        result => none,
+        errors => [Tag]
+    }.
+
+%%
+%% PidRef Operations
+%%
+
+-spec get_pid_ref() -> maybe_pid_ref().
+get_pid_ref() ->
+    couch_srt_util:get_pid_ref().
+
+-spec get_pid_ref(Rctx :: rctx()) -> pid_ref().
+get_pid_ref(Rctx) ->
+    couch_srt_util:get_pid_ref(Rctx).
+
+-spec set_pid_ref(PidRef :: pid_ref()) -> pid_ref().
+set_pid_ref(PidRef) ->
+    couch_srt_util:set_pid_ref(PidRef).
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    couch_srt_server:create_pid_ref().
+
+-spec destroy_pid_ref() -> maybe_pid_ref().
+destroy_pid_ref() ->
+    destroy_pid_ref(get_pid_ref()).
+
+%%destroy_pid_ref(undefined) ->
+%%    undefined;
+-spec destroy_pid_ref(PidRef :: maybe_pid_ref()) -> maybe_pid_ref().
+destroy_pid_ref(_PidRef) ->
+    erase(?PID_REF).
+
+%%
+%% Context Lifecycle API
+%%
+
+-spec create_worker_context(From, MFA, Nonce) -> pid_ref() | false when
+    From :: pid_ref(), MFA :: mfa(), Nonce :: nonce().
+create_worker_context(From, {M, F, _A}, Nonce) ->
+    case is_enabled() of
+        true ->
+            Type = #rpc_worker{from = From, mod = M, func = F},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_coordinator_context(Httpd, Path) -> pid_ref() | false when
+    Httpd :: #httpd{}, Path :: list().
+create_coordinator_context(#httpd{method = Verb, nonce = Nonce}, Path0) ->
+    case is_enabled() of
+        true ->
+            Path = list_to_binary([$/ | Path0]),
+            Type = #coordinator{method = Verb, path = Path},
+            create_context(Type, Nonce);
+        false ->
+            false
+    end.
+
+-spec create_context(Type :: rctx_type(), Nonce :: term()) -> pid_ref() | false.
+create_context(Type, Nonce) ->
+    Rctx = couch_srt_server:new_context(Type, Nonce),
+    PidRef = get_pid_ref(Rctx),
+    set_pid_ref(PidRef),
+    try
+        couch_srt_util:put_delta_a(Rctx),
+        couch_srt_util:put_updated_at(Rctx),
+        couch_srt_server:create_resource(Rctx),
+        couch_srt_logger:track(Rctx),
+        PidRef
+    catch
+        _:_ ->
+            couch_srt_server:destroy_resource(PidRef),
+            %% calling destroy_context(PidRef) clears the tracker too
+            destroy_context(PidRef),
+            false
+    end.
+
+-spec set_context_dbname(DbName :: binary()) -> boolean().
+set_context_dbname(DbName) ->
+    set_context_dbname(DbName, get_pid_ref()).
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: binary(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_dbname(DbName, PidRef).
+
+-spec set_context_handler_fun(Handler) -> boolean() when
+    Handler :: function() | {atom(), atom()}.
+set_context_handler_fun(Handler) ->
+    set_context_handler_fun(Handler, get_pid_ref()).
+
+-spec set_context_handler_fun(Handler, PidRef) -> boolean() when
+    Handler :: function() | {atom(), atom()}, PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun(Fun, PidRef) when is_function(Fun) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            FProps = erlang:fun_info(Fun),
+            Mod = proplists:get_value(module, FProps),
+            Func = proplists:get_value(name, FProps),
+            set_context_handler_fun({Mod, Func}, PidRef)
+    end;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case is_enabled() of
+        false ->
+            false;
+        true ->
+            couch_srt_server:set_context_handler_fun({Mod, Func}, PidRef)
+    end.
+
+%% @equiv set_context_username(User, get_pid_ref())
+set_context_username(User) ->
+    set_context_username(User, get_pid_ref()).
+
+-spec set_context_username(User, PidRef) -> boolean() when
+    User :: null | undefined | #httpd{} | #user_ctx{} | binary(),
+    PidRef :: maybe_pid_ref().
+set_context_username(null, _) ->
+    false;
+set_context_username(_, undefined) ->
+    false;
+set_context_username(#httpd{user_ctx = Ctx}, PidRef) ->
+    set_context_username(Ctx, PidRef);
+set_context_username(#user_ctx{name = Name}, PidRef) ->
+    set_context_username(Name, PidRef);
+set_context_username(UserName, PidRef) ->
+    is_enabled() andalso couch_srt_server:set_context_username(UserName, PidRef).
+
+-spec destroy_context() -> ok.
+destroy_context() ->
+    destroy_context(get_pid_ref()).
+
+-spec destroy_context(PidRef :: maybe_pid_ref()) -> ok.
+destroy_context(undefined) ->
+    ok;
+destroy_context(PidRef) ->
+    %% Stopping the tracker clears the ets entry for PidRef on its way out
+    couch_srt_logger:stop_tracker(),
+    destroy_pid_ref(PidRef),
+    clear_pdict_markers(),
+    ok.
+
+-spec clear_pdict_markers() -> ok.
+clear_pdict_markers() ->
+    ok = lists:foreach(
+        fun
+            ({{csrt, _} = K, _V}) ->
+                erlang:erase(K);
+            (_) ->
+                ok
+        end,
+        erlang:get()
+    ).
+
+%%
+%% Public API
+%%
+
+-spec maybe_track_rexi_init_p({M, F, A}) -> couch_stats:response() when
+    M :: atom(), F :: atom(), A :: non_neg_integer().
+maybe_track_rexi_init_p({M, F, _A}) ->
+    Metric = [M, F, spawned],
+    case couch_srt:should_track_init_p(Metric) of
+        true -> couch_stats:increment_counter(Metric);
+        false -> ok
+    end.
+
+%% Only potentially track positive increments to counters
+-spec maybe_track_local_counter(any(), any()) -> ok.
+maybe_track_local_counter(Name, Val) when is_integer(Val) andalso Val > 0 ->
+    couch_srt:maybe_inc(Name, Val),
+    ok;
+maybe_track_local_counter(_, _) ->
+    ok.
+
+%% @equiv couch_srt_util:is_enabled().
+-spec is_enabled() -> boolean().
+is_enabled() ->
+    couch_srt_util:is_enabled().
+
+%% @equiv couch_srt_util:is_enabled_reporting().
+-spec is_enabled_reporting() -> boolean().
+is_enabled_reporting() ->
+    couch_srt_util:is_enabled_reporting().
+
+%% @equiv couch_srt_util:is_enabled_rpc_reporting().
+-spec is_enabled_rpc_reporting() -> boolean().
+is_enabled_rpc_reporting() ->
+    couch_srt_util:is_enabled_rpc_reporting().
+
+%% @equiv couch_srt_util:is_enabled_init_p().
+-spec is_enabled_init_p() -> boolean().
+is_enabled_init_p() ->
+    couch_srt_util:is_enabled_init_p().
+
+-spec get_resource() -> maybe_rctx().
+get_resource() ->
+    get_resource(get_pid_ref()).
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(PidRef) ->
+    couch_srt_server:get_resource(PidRef).
+
+%% Log a CSRT report if any filters match
+-spec maybe_report(ReportName :: string(), PidRef :: pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    couch_srt_logger:maybe_report(ReportName, PidRef).
+
+%% Direct report logic skipping should log filters
+-spec do_report(ReportName :: string(), PidRef :: pid_ref()) -> boolean().
+do_report(ReportName, PidRef) ->
+    couch_srt_logger:do_report(ReportName, get_resource(PidRef)).
+
+-spec to_json(Rctx :: maybe_rctx()) -> map() | null.
+to_json(undefined) ->
+    null;
+to_json(Rctx) ->
+    couch_srt_entry:to_json(Rctx).
+
+%%
+%% Stat Collection API
+%%
+
+-spec inc(Key :: rctx_field()) -> non_neg_integer().
+inc(Key) ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key);
+        false ->
+            0
+    end.
+
+-spec inc(Key :: rctx_field(), N :: non_neg_integer()) -> non_neg_integer().
+inc(Key, N) when is_integer(N) andalso N >= 0 ->
+    case is_enabled() of
+        true ->
+            couch_srt_server:inc(get_pid_ref(), Key, N);
+        false ->
+            0
+    end.
+
+-spec maybe_inc(Stat :: atom(), Val :: non_neg_integer()) -> non_neg_integer().
+maybe_inc(Stat, Val) ->
+    case maps:is_key(Stat, ?STATS_TO_KEYS) of
+        true ->
+            inc(maps:get(Stat, ?STATS_TO_KEYS), Val);
+        false ->
+            0
+    end.
+
+-spec should_track_init_p(Stat :: [atom()]) -> boolean().
+%% "Example to extend CSRT"
+%% should_track_init_p([fabric_rpc, foo, spawned]) ->
+%%    is_enabled_init_p();
+should_track_init_p([fabric_rpc, all_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, changes, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, get_all_security, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, map_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_doc, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, open_shard, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, reduce_view, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p([fabric_rpc, update_docs, spawned]) ->
+    is_enabled_init_p();
+should_track_init_p(_Metric) ->
+    false.
+
+-spec ioq_called() -> non_neg_integer().
+ioq_called() ->
+    inc(ioq_calls).
+
+%% we cannot yet use stats couchdb.query_server.*.ddoc_filter because those
+%% are collected in a dedicated process.
+%% TODO: funnel back stats from background worker processes to the RPC worker
+js_filtered(N) ->
+    inc(js_filter),
+    inc(js_filtered_docs, N).
+
+docs_written(N) ->
+    inc(docs_written, N).
+
+-spec accumulate_delta(Delta :: map() | undefined) -> ok.
+accumulate_delta(Delta) when is_map(Delta) ->
+    is_enabled() andalso couch_srt_server:update_counters(get_pid_ref(), Delta),
+    ok;
+accumulate_delta(undefined) ->
+    ok.
+
+-spec make_delta() -> maybe_delta().
+make_delta() ->
+    case is_enabled() of
+        false ->
+            undefined;
+        true ->
+            couch_srt_util:make_delta(get_pid_ref())
+    end.
+
+-spec rctx_delta(TA :: maybe_rctx(), TB :: maybe_rctx()) -> maybe_delta().
+rctx_delta(TA, TB) ->
+    couch_srt_util:rctx_delta(TA, TB).
+
+%%
+%% Aggregate Query API
+%%
+
+-spec active() -> [rctx()].
+active() ->
+    couch_srt_query:active().
+
+-spec active(Type :: json) -> [rctx()].
+active(Type) ->
+    couch_srt_query:active(Type).
+
+-spec active_coordinators() -> [coordinator_rctx()].
+active_coordinators() ->
+    couch_srt_query:active_coordinators().
+
+%% TODO: cleanup json logic here
+-spec active_coordinators(Type :: json) -> [coordinator_rctx()].
+active_coordinators(Type) ->
+    couch_srt_query:active_coordinators(Type).
+
+-spec active_workers() -> [rpc_worker_rctx()].
+active_workers() ->
+    couch_srt_query:active_workers().
+
+-spec active_workers(Type :: json) -> [rpc_worker_rctx()].
+active_workers(Type) ->
+    couch_srt_query:active_workers(Type).
+
+find_by_nonce(Nonce) ->
+    couch_srt_query:find_by_nonce(Nonce).
+
+find_by_pid(Pid) ->
+    couch_srt_query:find_by_pid(Pid).
+
+find_by_pidref(PidRef) ->
+    couch_srt_query:find_by_pidref(PidRef).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_query:find_workers_by_pidref(PidRef).
+
+-spec pid_ref_matchspec(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_matchspec(AttrName) ->
+    couch_srt_logger:pid_ref_matchspec(AttrName).
+
+-spec pid_ref_attrs(AttrName :: rctx_field()) -> term() | throw(any()).
+pid_ref_attrs(AttrName) ->
+    couch_srt_logger:pid_ref_attrs(AttrName).
+
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with couch_srt_logger:pid_ref_attrs/1, and
+%% returning on pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    couch_srt_logger:proc_window(AttrName, Num, Time).
+
+-spec query_matcher(MatcherName :: matcher_name()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName) ->
+    couch_srt_query:query_matcher(MatcherName).
+
+-spec query_matcher(MatcherName :: matcher_name(), Limit :: pos_integer()) ->
+    {ok, query_result()}
+    | {error, any()}.
+query_matcher(MatcherName, Limit) ->
+    couch_srt_query:query_matcher(MatcherName, Limit).
+
+%%
+%% Delta API
+%%
+
+-spec add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+add_delta(T, Delta) ->
+    couch_srt_util:add_delta(T, Delta).
+
+-spec extract_delta(T :: term_delta()) -> {term(), maybe_delta()}.
+extract_delta(T) ->
+    couch_srt_util:extract_delta(T).
+
+-spec get_delta() -> tagged_delta().
+get_delta() ->
+    couch_srt_util:get_delta(get_pid_ref()).
+
+-spec maybe_add_delta(T :: term()) -> term_delta().
+maybe_add_delta(T) ->
+    couch_srt_util:maybe_add_delta(T).
+
+-spec maybe_add_delta(T :: term(), Delta :: maybe_delta()) -> term_delta().
+maybe_add_delta(T, Delta) ->
+    couch_srt_util:maybe_add_delta(T, Delta).
+
+%%
+%% Query API functions
+%%
+%%
+
+%% @doc Construct query from the expressions.
+%% There are following types of expressions allowed in the query.
+%%   <li>group_by/1 @see group_by/1</li>
+%%   <li>group_by/2 @see group_by/1</li>
+%%   <li>sort_by/1 @see sort_by/1</li>
+%%   <li>count_by/1 @see count_by/1</li>
+%%   <li>options/1 @see options/1</li>
+%%   <li>from/1 @see from/1</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec query(QueryExpression :: [query_expression()]) ->
+    query() | {error, any()}.
+query(QueryExpression) ->
+    couch_srt_query:query(QueryExpression).
+
+%% @doc Specify the matcher to use for the query.
+%% If atom 'all' is used then all entries would be in the scope of the query.
+%% Also the use of 'all' makes the query 'unsafe'. Because it scans through all entries
+%% and can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    from("docs_read")
+%% ]),
+%% </code>
+%% @end
+-spec from(MatcherNameOrAll :: string() | all) ->
+    query_expression() | {error, any()}.
+from(MatcherNameOrAll) ->
+    couch_srt_query:from(MatcherNameOrAll).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+group_by(AggregationKeys) ->
+    couch_srt_query:group_by(AggregationKeys).
+
+%% @doc Request 'group_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    group_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec group_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+group_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:group_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname])
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+sort_by(AggregationKeys) ->
+    couch_srt_query:sort_by(AggregationKeys).
+
+%% @doc Request 'sort_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    sort_by([username, dbname], ioq_calls)
+%% ]),
+%% </code>
+%% @end
+-spec sort_by(AggregationKeys, ValueKey) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()],
+    ValueKey ::
+        binary()
+        | rctx_field().
+sort_by(AggregationKeys, ValueKey) ->
+    couch_srt_query:sort_by(AggregationKeys, ValueKey).
+
+%% @doc Request 'count_by' aggregation of results.
+%% <code>
+%% Q = query([
+%%    ...
+%%    count_by(username)
+%% ]),
+%% </code>
+%% @end
+-spec count_by(AggregationKeys) ->
+    query_expression() | {error, any()}
+when
+    AggregationKeys ::
+        binary()
+        | rctx_field()
+        | [binary()]
+        | [rctx_field()].
+count_by(AggregationKeys) ->
+    couch_srt_query:count_by(AggregationKeys).
+
+%% @doc Construct 'options' query expression.
+%% There are following types of expressions allowed in the query.
+%%   <li>unlimited/0 @see unlimited/0 (cannot be used with 'with_limit/1')</li>
+%%   <li>with_limit/1 @see with_limit/1 (cannot be used with 'unlimited/0')</li>
+%% The order of expressions doesn't matter.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      ...
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec options([query_option()]) ->
+    query_expression() | {error, any()}.
+options(OptionsExpression) ->
+    couch_srt_query:options(OptionsExpression).
+
+%% @doc Enable unlimited number of results from the query.
+%% The use of 'unlimited' makes the query 'unsafe'. Because it can return many matching rows.
+%% Unsafe queries can only be run using 'unsafe_run/1'.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      unlimited()
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec unlimited() ->
+    query_expression().
+unlimited() ->
+    couch_srt_query:unlimited().
+
+%% @doc Set limit on number of results returned from the query.
+%% The construction of the query fail if the 'limit' is greater than
+%% allowed for this cluster.
+%% <code>
+%% Q = query([
+%%    ...
+%%    options([
+%%      with_limit(100)
+%%    ])
+%% ]),
+%% </code>
+%% @end
+-spec with_limit(Limit :: pos_integer()) ->
+    query_expression() | {error, any()}.
+with_limit(Limit) ->
+    couch_srt_query:with_limit(Limit).
+
+%% @doc Executes provided query. Only 'safe' queries can be executed using 'run'.
+%% The query considered 'unsafe' if any of the conditions bellow are met:
+%%   <li>Query uses 'unlimited/0'</li>
+%%   <li>Query uses 'from(all)'</li>
+%% <code>
+%% Q = query([
+%%    from("docs_read"),
+%%    group_by(username, dbname, ioq_calls),
+%%    options([
+%%       with_limit(10)
+%%    ])
+%% ]),
+%% run(Q)
+%% </code>
+%% @end
+-spec run(query()) ->
+    {ok, [{aggregation_key(), pos_integer()}]}
+    | {limit, [{aggregation_key(), pos_integer()}]}.
+run(Query) ->

Review Comment:
   There query API unify different aggregation APIs under one roof as a declarative DSL. Initial non-DSL version had own functions for each aggregation (as you seem to suggest). The problem was that in that implementation the closures were send to another node via erlang distribution protocol. Also since the functions were separate, validation of arguments for them was harder (the internal representation details were used in http API handler code). For that reason the validation (in initial version) was omitted, the let it crash approach was used. The invalid query was crashing when it started the iteration over ets table. To fix the argument validation story and avoid sending the closures I designed the query language which resembles the SQL. With the idea to use it for:
    
   1. More capable HTTP API query endpoint
   2. A way to pre-define often used queries in `default.ini` (Although at that point the `CSRT` config would be too big to my taste, so I would prefer a separate file).  
   3. Use it in couch_debug helpers
   4. This would be useful in a modernized `weatherreport` replacement app
   
   These four ideas are for the future and subject to follow up PRs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2236591607


##########
src/chttpd/src/chttpd_httpd_handlers.erl:
##########
@@ -20,6 +20,7 @@ url_handler(<<"_utils">>) -> fun chttpd_misc:handle_utils_dir_req/1;
 url_handler(<<"_all_dbs">>) -> fun chttpd_misc:handle_all_dbs_req/1;
 url_handler(<<"_dbs_info">>) -> fun chttpd_misc:handle_dbs_info_req/1;
 url_handler(<<"_active_tasks">>) -> fun chttpd_misc:handle_task_status_req/1;
+url_handler(<<"_active_resources">>) -> fun couch_srt_httpd:handle_resource_status_req/1;

Review Comment:
   I've got the documentation reworked into the formal Sphinx docs and now we've got a CSRT working set of docs to extend from, I'm hoping @iilyak can update those docs with his HTTP API updates. I was also wondering if there's a simple way to get the inline docs from the `couch_srt_query` module, as those have a lot of great info, where do those end up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3127712922

   Alright I've gone through and addressed most things:
   
   * I've reworked the docs to with more details on how/where/when/why to utilize CSRT and also moved them into the Sphinx docs in: 828667e3d as well as introducing a new example or two
   * In eb92a4d5a I reworked `csrt_query:group_by` to use the more efficient `ets:select` as well as to utilize that to enforce the `query_cardinality_limit()` constraints which I also updated to be more clear in that commit. I also fixed a small bug in `csrt_query:options/1`
   * I added default Logger Matchers `all_coordinators` and `all_rpc_workers` in 00a02af01 so we'd have an easy and dedicated way to toggle on logging of all coordinate process lifecycle reports, and if so desired, the same with `all_rpc_workers`, but the other impact of this change is that we can now easily aggregate on only coordinators or workers when querying against the HTTP and query API
   * updated `couch_log_formatter:format_meta` to ignore `null` values so we don't end up with report entries like `dbname="null"`
   * cleaned up various test cases based on PR beedback in 1380b09fc
   * various other cleanup bits


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2257227004


##########
src/couch_srt/src/couch_srt_httpd.erl:
##########
@@ -0,0 +1,157 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch_srt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+handle_resource_status_req(
+    #httpd{method = 'POST', path_parts = [<<"_active_resources">>, <<"_match">>, MatcherNameBin]} =
+        Req
+) ->
+    chttpd:validate_ctype(Req, "application/json"),
+    {JsonProps} = chttpd:json_body_obj(Req),
+    GroupBy = couch_util:get_value(<<"group_by">>, JsonProps),
+    SortBy = couch_util:get_value(<<"sort_by">>, JsonProps),
+    CountBy = couch_util:get_value(<<"count_by">>, JsonProps),
+    MatcherName = binary_to_list(MatcherNameBin),
+    {AggregationKeys, Query} =
+        case {GroupBy, SortBy, CountBy} of
+            {undefined, undefined, {Props}} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                {Keys, couch_srt:query([couch_srt:from(MatcherName), couch_srt:count_by(Keys)])};
+            {undefined, {Props}, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:sort_by(Keys, CounterKey)
+                    ])};
+            {{Props}, undefined, undefined} ->
+                Keys = couch_util:get_value(<<"aggregate_keys">>, Props),
+                CounterKey = couch_util:get_value(<<"counter_key">>, Props),
+                {Keys,
+                    couch_srt:query([
+                        couch_srt:from(MatcherName), couch_srt:group_by(Keys, CounterKey)
+                    ])};
+            {_, _, _} ->
+                throw({bad_request, <<"Multiple aggregations are not supported">>})
+        end,
+    case Query of
+        {error, Reason} ->
+            send_error(Req, Reason);
+        Q ->
+            JSON = to_json(AggregationKeys, couch_srt:rpc_run(Q)),
+            send_json(Req, JSON)
+    end;
+handle_resource_status_req(#httpd{path_parts = [<<"_active_resources">>]} = Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD");
+handle_resource_status_req(Req) ->
+    ok = chttpd:verify_is_server_admin(Req),
+    send_method_not_allowed(Req, "GET,HEAD,POST").
+
+to_json(AggregationKeys, Results) ->
+    lists:map(fun(E) -> node_reply_to_json(AggregationKeys, E) end, Results).
+
+node_reply_to_json(_AggregationKeys, #{node := Node, result := none, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => none,
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    };
+node_reply_to_json(AggregationKeys, #{node := Node, result := Result, errors := Errors}) ->
+    #{
+        node => atom_to_binary(Node),
+        result => aggregation_result_to_json(AggregationKeys, Result),
+        errors => lists:map(fun erlang:atom_to_list/1, Errors)
+    }.
+
+encode_key(AggregationKeys, Key) ->
+    maps:from_list(lists:zip(AggregationKeys, tuple_to_list(Key))).
+
+-spec aggregation_result_to_json(AggregationKeys :: binary() | [binary()], Map :: query_result()) ->
+    json_spec(#{
+        value => non_neg_integer(),
+        key => #{
+            username => string(),

Review Comment:
   I added the json_string() type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2220346513


##########
src/couch_stats/src/csrt_httpd.erl:
##########
@@ -0,0 +1,169 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_httpd).
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("csrt.hrl").
+
+-export([handle_resource_status_req/1]).
+
+-import(
+    chttpd,
+    [
+        send_json/2, send_json/3,
+        send_method_not_allowed/2
+    ]
+).
+
+-import(
+    csrt,
+    [
+        query/1,
+        from/1,
+        group_by/1,
+        group_by/2,
+        sort_by/1,
+        sort_by/2,
+        count_by/1,
+        options/1,
+        unlimited/0,
+        with_limit/1,
+
+        run/1
+    ]
+).

Review Comment:
   That could make sense in general but I think it can still use some cleanup in this case.
   
    For instance I see
      `{Keys, csrt:query([from(MatcherName), count_by(Keys)])};` 
   but then also
     ` {Keys, query([from(MatcherName), sort_by(Keys, CounterKey)])};` 
   
   *  `couch_util:value(...)` doesn't have the shortened imported format but seems just as verbose as csrt:from(...)
   
   * `with_limit/1` doesn't seem to be used but is imported
   
   There 17 lines of imports but only 3 lines of `query(...)` calls so it seems overkill in this module as at least?
   
   Sometimes to avoid imports if we're really repeating the same call too often we can make a few short local functions. Say, if we call `erlang:monotonic_time()` then could make local one-line function called `tnow() -> erlang:monotonic_time()` then we can call that 20 or 30 times and it looks a bit better.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221034383


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true

Review Comment:
   These `[csrt.init_p]` values are awkward... and unfortunately _do_ need to be set here by default. All of the other values in this `default.ini` have default values to be set to and disabled, but these need to be set to true to have a lookup enabled for these settings and others.
   
   The original idea with `[csrt.init_p]` is that this would prevent the counter increments from triggering ever to prevent the tracking space for those counters to be enabled, but the folsom should_notify stuff isn't used directly anymore and besides the metrics are being picked up from the declaration of `stats_descriptions.cfg`.
   
   We need a positive lookup table for these values to return true to indicate they are they desired subset of operations spawned in `rexi_server:init_p/3`, as otherwise we'd create metrics for _all_ `fabric_rpc:*` functions. Alternatively, this could be a static map in the `csrt.hrl` file, but we need a way to add more to these later. I'm admittedly not a huge fan of this particular aspect of the init_p logic, but like I said, we need a positive lookup table somewhere.
   
   As for the double underscore, we've talked about this in the last PR, there's _zero_ precedent for the use of `:` in the codebase, and the only keyword that isn't `[a-z0-9_]` is `||*` in `partitioned||*`, eg:
   
   ```
   (! 11775)-> grep '^[a-z;].* = ' rel/overlay/etc/default.ini | sed 's/^\([a-z;][^ ]*\) =.*$/\1/' | grep -v ' ' | grep '[^a-z0-9_;]'
   partitioned||*
   ```
   
   and I was not able to find documentation that `:` is valid for default.ini, so I did not use that. Because of the `||` use, I used it in IOQ in the past https://github.com/apache/couchdb-ioq/blob/main/include/ioq.hrl#L26 but found `||` to be ugly, so I went with double underscore.
   
   Anyways... just let's just remove this entirely, the dynamic config toggle approach with the specific metrics didn't workout as I had wanted, so let's just go back to hardcoding these directly, that'll simplify this, eliminate the ini section entirely, and also get rid of the ugly string conversion in `csrt_util:fabric_conf_key/1`.
   
   Here's the patch to hardcode it, and I'll drop out the other components in a bit:
   
   ```
   (! 11789)-> git diff src/couch_stats/src/csrt.erl
   diff --git a/src/couch_stats/src/csrt.erl b/src/couch_stats/src/csrt.erl
   index 6c098bfea..3532a3b2b 100644
   --- a/src/couch_stats/src/csrt.erl
   +++ b/src/couch_stats/src/csrt.erl
   @@ -431,8 +431,25 @@ maybe_inc(Stat, Val) ->
        end.
    
    -spec should_track_init_p(Stat :: [atom()]) -> boolean().
   -should_track_init_p([Mod, Func, spawned]) ->
   -    is_enabled_init_p() andalso csrt_util:should_track_init_p(Mod, Func);
   +%% "Example to extend CSRT"
   +%% should_track_init_p([fabric_rpc, foo, spawned]) ->
   +%%    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, all_docs, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, changes, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, get_all_security, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, map_view, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, open_doc, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, open_shard, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, reduce_view, spawned]) ->
   +    is_enabled_init_p();
   +should_track_init_p([fabric_rpc, update_docs, spawned]) ->
   +    is_enabled_init_p();
    should_track_init_p(_Metric) ->
        false.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221281060


##########
src/couch_stats/src/csrt_entry.erl:
##########
@@ -0,0 +1,244 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_entry).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-export([
+    value/2,
+    key/1,
+    from_map/1,
+    record_info/0
+]).
+
+%% JSON Conversion API
+-export([
+    convert_type/1,
+    convert_pidref/1,
+    convert_pid/1,
+    convert_ref/1,
+    convert_string/1,
+    to_json/1
+]).
+
+-spec value(rctx_field(), #rctx{}) -> any().
+
+value(pid_ref, #rctx{pid_ref = Val}) -> Val;
+value(nonce, #rctx{nonce = Val}) -> Val;
+value(type, #rctx{type = Val}) -> convert_type(Val);
+value(dbname, #rctx{dbname = Val}) -> Val;
+value(username, #rctx{username = Val}) -> Val;
+value(db_open, #rctx{db_open = Val}) -> Val;
+value(docs_read, #rctx{docs_read = Val}) -> Val;
+value(docs_written, #rctx{docs_written = Val}) -> Val;
+value(rows_read, #rctx{rows_read = Val}) -> Val;
+value(changes_returned, #rctx{changes_returned = Val}) -> Val;
+value(ioq_calls, #rctx{ioq_calls = Val}) -> Val;
+value(js_filter, #rctx{js_filter = Val}) -> Val;
+value(js_filtered_docs, #rctx{js_filtered_docs = Val}) -> Val;
+value(get_kv_node, #rctx{get_kv_node = Val}) -> Val;
+value(get_kp_node, #rctx{get_kp_node = Val}) -> Val;
+value(started_at, #rctx{started_at = Val}) -> Val;
+value(updated_at, #rctx{updated_at = Val}) -> Val.
+
+-spec key(BinKey :: binary() | string() | atom()) ->
+    Key ::
+        rctx_field()
+        | {error, Reason :: any()}.
+
+key(Key) when is_atom(Key) ->
+    key_from_atom(Key);
+key(Key) when is_binary(Key) ->
+    key_from_binary(Key);
+key(Key) when is_list(Key) ->
+    case key_from_binary(list_to_binary(Key)) of
+        {error, {invalid_key, _Key}} ->
+            {error, {invalid_key, Key}};
+        Res ->
+            Res
+    end;
+key(Other) ->
+    key_error(Other).
+
+key_from_atom(pid_ref) -> pid_ref;
+key_from_atom(nonce) -> nonce;
+key_from_atom(type) -> type;
+key_from_atom(dbname) -> dbname;
+key_from_atom(username) -> username;
+key_from_atom(db_open) -> db_open;
+key_from_atom(docs_read) -> docs_read;
+key_from_atom(rows_read) -> rows_read;
+key_from_atom(changes_returned) -> changes_returned;
+key_from_atom(ioq_calls) -> ioq_calls;
+key_from_atom(js_filter) -> js_filter;
+key_from_atom(js_filtered_docs) -> js_filtered_docs;
+key_from_atom(get_kv_node) -> get_kv_node;
+key_from_atom(get_kp_node) -> get_kp_node;
+key_from_atom(Other) -> key_error(Other).
+
+key_from_binary(<<"pid_ref">>) -> pid_ref;
+key_from_binary(<<"nonce">>) -> nonce;
+key_from_binary(<<"type">>) -> type;
+key_from_binary(<<"dbname">>) -> dbname;
+key_from_binary(<<"username">>) -> username;
+key_from_binary(<<"db_open">>) -> db_open;
+key_from_binary(<<"docs_read">>) -> docs_read;
+key_from_binary(<<"rows_read">>) -> rows_read;
+key_from_binary(<<"changes_returned">>) -> changes_returned;
+key_from_binary(<<"ioq_calls">>) -> ioq_calls;
+key_from_binary(<<"js_filter">>) -> js_filter;
+key_from_binary(<<"js_filtered_docs">>) -> js_filtered_docs;
+key_from_binary(<<"get_kv_node">>) -> get_kv_node;
+key_from_binary(<<"get_kp_node">>) -> get_kp_node;
+key_from_binary(Other) -> key_error(Other).

Review Comment:
   Same thing as above with `key_from_atom`, yes, this is verbosity, but it's well encapsulated into specific functions to hide that verbosity, and done in a clean way that avoids having to do dynamic runtime conversions.
   
   Again, I'd prefer to see parse transforms on top of the `#rctx{}` record to generate all of these functions, but for a first pass we've got hardcoded conversion functions to efficiently handle the logic, as well as various use of static analysis to ensure that the `rctx_field()`'s are picked up as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#issuecomment-3105033930

   Okay over in 38608f7cd I've dropped the extraneous `abs` usage, cleaned up the gen_server callbacks, dropped the `unlink` call, and cleaned up some naming discrepancies, plus I've added additional documentation clarifying that the default logger matchers are for matching against any requests in CouchDB as opposed to the "dbnames_io" matchers being targeted to a specific database name, and mentioned the tradeoff of losing some of the granularity of matching against particular dimensions. Alternatively, we could add `dbnames_ioq` matcher for creating a matcher on a dbname with a particular IOQ threshold, but because we can't easily chain these definition's we would need to make dedicated matchers for _all_ of the desired combinations, eg `dbnames_docs_read`, `dbnames_rows_read`, etc. The combinatorics become even more problematic when you want to express matchers like "match on db foo for changes requests that have induced more than 1000 IOQ calls and more than 100 Javascript filter
  invocations".
   
   You can actually create that matcher now, but it needs to be registered directly by way of remsh, for example, the follow dynamically creates a CSRT logger matcher that satisfies the constraint `"match on db foo for changes requests that have induced more than 1000 IOQ calls and more than 100 Javascript filter invocations".`:
   
   ```erlang
   (node1@127.0.0.1)16> rr(csrt_server).
   [coordinator,rctx,rpc_worker,st]
   (node1@127.0.0.1)17> ets:fun2ms(fun(#rctx{dbname = <<"foo">>, type=#coordinator{mod='chttpd_db', func='handle_changes_req'}, ioq_calls=IC, js_filter=JF}=R) when IC > 1000 andalso JF > 100 -> R end).
   [{#rctx{started_at = '_',updated_at = '_',pid_ref = '_',
           nonce = '_',
           type = #coordinator{mod = chttpd_db,
                               func = handle_changes_req,method = '_',path = '_'},
           dbname = <<"foo">>,username = '_',db_open = '_',
           docs_read = '_',docs_written = '_',rows_read = '_',
           changes_returned = '_',ioq_calls = '$1',js_filter = '$2',
           js_filtered_docs = '_',get_kv_node = '_',get_kp_node = '_'},
     [{'andalso',{'>','$1',1000},{'>','$2',100}}],
     ['$_']}]
   (node1@127.0.0.1)18> csrt_logger:register_matcher("custom_foo", ets:fun2ms(fun(#rctx{dbname = <<"food_db', func='handle_changes_req'}, ioq_calls=IC, js_filter=JF}=R) when IC > 1000 andalso JF > 100 -> R end)).
   ok
   ```
   
   That'll dynamically compile the matchspec and push it out by way of persistent_term to be picked up by the tracker pids to decide whether or not to generate a process lifecycle report.
   
   The tricky bit is mapping `ets:fun2ms(fun(#rctx{dbname = <<"foo">>, type=#coordinator{mod='chttpd_db', func='handle_changes_req'}, ioq_calls=IC, js_filter=JF}=R) when IC > 1000 andalso JF > 100 -> R end)` to something we can express in `default.ini`. The `ets:fun2ms` transform is a great tool that facilitates declaratively constructing these matchspecs that allow us to efficiently query the ets tracking table, while also re-using these same matchers directly against a given `#rctx{}` to filter requests to log, however, its use of parse transforms makes it difficult to iteratively and programmatically construct complex pattern match statements.
   
   I would love to see a simple translation layer that basically lets us use Mango syntax for declaring these filters, as that would make it much easier to express within the ini files, but also it would allow us to dynamically construct the logger matcher specs on the fly for a given HTTP request, eg you could `POST` Mango spec but with fields in `#rctx{}` to then query the ets table. If we get to where we have the expressiveness of something like Mango query syntax to define the logger matchers, then we can replace most of these default matchers with better more targeted matchers while also providing an HTTP query API that can dynamically generate these ets matchspecs for efficient querying and aggregating.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221237408


##########
src/couch_stats/src/csrt_logger.erl:
##########
@@ -0,0 +1,592 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_logger).
+
+%% Process lifetime logging api
+-export([
+    get_tracker/0,
+    log_process_lifetime_report/1,
+    put_tracker/1,
+    stop_tracker/0,
+    stop_tracker/1,
+    track/1,
+    tracker/1
+]).
+
+%% Raw API that bypasses is_enabled checks
+-export([
+    do_lifetime_report/1,
+    do_status_report/1,
+    do_report/2,
+    maybe_report/2,
+    should_truncate_reports/0
+]).
+
+%% gen_server callbacks
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+%% Config update subscription API
+-export([
+    subscribe_changes/0,
+    handle_config_change/5,
+    handle_config_terminate/3
+]).
+
+%% Matchers
+-export([
+    deregister_matcher/1,
+    find_all_matches/2,
+    find_matches/2,
+    get_matcher/1,
+    get_matchers/0,
+    get_registered_matchers/0,
+    is_match/1,
+    is_match/2,
+    matcher_on_dbname/1,
+    matcher_on_docs_read/1,
+    matcher_on_docs_written/1,
+    matcher_on_rows_read/1,
+    matcher_on_changes_processed/1,
+    matcher_on_ioq_calls/1,
+    matcher_on_nonce/1,
+    matcher_on_long_reqs/1,
+    register_matcher/2,
+    reload_matchers/0
+]).
+
+%% Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {
+    registered_matchers = #{}
+}).
+
+-spec track(Rctx :: rctx()) -> pid().
+track(#rctx{pid_ref = PidRef}) ->
+    case get_tracker() of
+        undefined ->
+            Pid = spawn(?MODULE, tracker, [PidRef]),
+            put_tracker(Pid),
+            Pid;
+        Pid when is_pid(Pid) ->
+            Pid
+    end.
+
+-spec tracker(PidRef :: pid_ref()) -> ok.
+tracker({Pid, _Ref} = PidRef) ->
+    MonRef = erlang:monitor(process, Pid),
+    receive
+        stop ->
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok;
+        {'DOWN', MonRef, _Type, _0DPid, _Reason0} ->
+            %% TODO: should we pass reason to log_process_lifetime_report?
+            %% Reason = case Reason0 of
+            %%     {shutdown, Shutdown0} ->
+            %%         Shutdown = atom_to_binary(Shutdown0),
+            %%         <<"shutdown: ", Shutdown/binary>>;
+            %%     Reason0 ->
+            %%         Reason0
+            %% end,
+            %% TODO: should we send the induced work delta to the coordinator?
+            log_process_lifetime_report(PidRef),
+            csrt_server:destroy_resource(PidRef),
+            ok
+    end.
+
+-spec register_matcher(Name, MSpec) -> ok | {error, badarg} when
+    Name :: string(), MSpec :: ets:match_spec().
+register_matcher(Name, MSpec) ->
+    gen_server:call(?MODULE, {register, Name, MSpec}, infinity).
+
+-spec deregister_matcher(Name :: string()) -> ok.
+deregister_matcher(Name) ->
+    gen_server:call(?MODULE, {deregister, Name}).
+
+-spec log_process_lifetime_report(PidRef :: pid_ref()) -> ok.
+log_process_lifetime_report(PidRef) ->
+    case csrt_util:is_enabled() andalso csrt_util:is_enabled_reporting() of
+        true ->
+            maybe_report("csrt-pid-usage-lifetime", PidRef);
+        false ->
+            ok
+    end.
+
+%% Return a subset of Matchers for each Matcher that matches on Rctxs
+-spec find_matches(Rctxs :: [rctx()], Matchers :: matchers()) -> matchers().
+find_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    Rctxs1 =
+        case csrt_util:is_enabled_rpc_reporting() of
+            true ->
+                Rctxs;
+            false ->
+                [Rctx || #rctx{type = #coordinator{}} = Rctx <- Rctxs]
+        end,
+    maps:filter(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            (catch ets:match_spec_run(Rctxs1, CompMSpec)) =/= []
+        end,
+        Matchers
+    ).
+
+%% Return a Map of #{MatcherName => SRctxs :: rctxs()} for all MatcherName => Matcher
+%% in Matchers where SRctxs is the subset of Rctxs matched by the given Matcher
+-spec find_all_matches(Rctxs :: rctxs(), Matchers :: matchers()) -> matcher_matches().
+find_all_matches(Rctxs, Matchers) when is_list(Rctxs) andalso is_map(Matchers) ->
+    maps:map(
+        fun(_Name, {_MSpec, CompMSpec}) ->
+            try
+                ets:match_spec_run(Rctxs, CompMSpec)
+            catch
+                _:_ ->
+                    []
+            end
+        end,
+        Matchers
+    ).
+
+-spec reload_matchers() -> ok.
+reload_matchers() ->
+    ok = gen_server:call(?MODULE, reload_matchers, infinity).
+
+-spec get_matchers() -> matchers().
+get_matchers() ->
+    persistent_term:get(?MATCHERS_KEY, #{}).
+
+-spec get_matcher(Name :: matcher_name()) -> maybe_matcher().
+get_matcher(Name) ->
+    maps:get(Name, get_matchers(), undefined).
+
+-spec get_registered_matchers() -> matchers().
+get_registered_matchers() ->
+    gen_server:call(?MODULE, get_registered_matchers, infinity).
+
+-spec is_match(Rctx :: maybe_rctx()) -> boolean().
+is_match(undefined) ->
+    false;
+is_match(#rctx{} = Rctx) ->
+    is_match(Rctx, get_matchers()).
+
+-spec is_match(Rctx :: maybe_rctx(), Matchers :: matchers()) -> boolean().
+is_match(undefined, _Matchers) ->
+    false;
+is_match(_Rctx, undefined) ->
+    false;
+is_match(#rctx{} = Rctx, Matchers) when is_map(Matchers) ->
+    maps:size(find_matches([Rctx], Matchers)) > 0.
+
+%% Generate a report for the Rctx if it triggers an active Matcher
+-spec maybe_report(ReportName :: string(), PidRef :: maybe_pid_ref()) -> ok.
+maybe_report(ReportName, PidRef) ->
+    Rctx = csrt_server:get_resource(PidRef),
+    case is_match(Rctx) of
+        true ->
+            do_report(ReportName, Rctx),
+            ok;
+        false ->
+            ok
+    end.
+
+%% Whether or not to remove zero value fields from reports to save on volume
+-spec should_truncate_reports() -> boolean().
+should_truncate_reports() ->
+    config:get_boolean(?CSRT, "should_truncate_reports", true).
+
+-spec do_lifetime_report(Rctx :: rctx()) -> boolean().
+do_lifetime_report(Rctx) ->
+    do_report("csrt-pid-usage-lifetime", Rctx).
+
+-spec do_status_report(Rctx :: rctx()) -> boolean().
+do_status_report(Rctx) ->
+    do_report("csrt-pid-usage-status", Rctx).
+
+-spec do_report(ReportName :: string(), Rctx :: rctx()) -> boolean().
+do_report(ReportName, #rctx{} = Rctx) ->
+    JRctx =
+        case {should_truncate_reports(), csrt_entry:to_json(Rctx)} of
+            {true, JRctx0} ->
+                maps:filter(fun(_K, V) -> V > 0 end, JRctx0);
+            {false, JRctx0} ->
+                JRctx0
+        end,
+    couch_log:report(ReportName, JRctx).
+
+%%
+%% Process lifetime logging api
+%%
+
+-spec get_tracker() -> maybe_pid().
+get_tracker() ->
+    get(?TRACKER_PID).
+
+-spec put_tracker(Pid :: pid()) -> maybe_pid().
+put_tracker(Pid) when is_pid(Pid) ->
+    put(?TRACKER_PID, Pid).
+
+-spec stop_tracker() -> ok.
+stop_tracker() ->
+    stop_tracker(get_tracker()).
+
+-spec stop_tracker(Pid :: maybe_pid()) -> ok.
+stop_tracker(undefined) ->
+    ok;
+stop_tracker(Pid) when is_pid(Pid) ->
+    Pid ! stop,
+    ok.
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    St = #st{},
+    ok = initialize_matchers(St#st.registered_matchers),
+    ok = subscribe_changes(),
+    {ok, St}.
+
+handle_call({register, Name, MSpec}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case add_matcher(Name, MSpec, RMatchers) of
+        {ok, RMatchers1} ->
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}};
+        {error, {invalid_ms, _, _}} = Error ->
+            {reply, Error, St}
+    end;
+handle_call({deregister, Name}, _From, #st{registered_matchers = RMatchers} = St) ->
+    case maps:is_key(Name, RMatchers) of
+        false ->
+            {reply, {error, missing_matcher}, St};
+        true ->
+            RMatchers1 = maps:remove(Name, RMatchers),
+            ok = initialize_matchers(RMatchers1),
+            {reply, ok, St#st{registered_matchers = RMatchers1}}
+    end;
+handle_call(reload_matchers, _From, St) ->
+    couch_log:warning("Reloading persistent term matchers", []),
+    ok = initialize_matchers(St#st.registered_matchers),
+    {reply, ok, St};
+handle_call(get_registered_matchers, _From, St) ->
+    {reply, St#st.registered_matchers, St};
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.
+
+handle_info(restart_config_listener, State) ->
+    ok = subscribe_changes(),
+    {noreply, State};
+handle_info(_Msg, St) ->
+    {noreply, St}.
+
+%%
+%% Matchers
+%%
+
+-spec matcher_on_dbname(DbName :: dbname()) -> ets:match_spec().
+matcher_on_dbname(DbName) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(#rctx{dbname = DbName1} = R) when DbName =:= DbName1 -> R end).
+
+-spec matcher_on_dbname_io_threshold(DbName, Threshold) -> ets:match_spec() when
+    DbName :: dbname(), Threshold :: pos_integer().
+matcher_on_dbname_io_threshold(DbName, Threshold) when
+    is_binary(DbName)
+->
+    ets:fun2ms(fun(
+        #rctx{
+            dbname = DbName1,
+            ioq_calls = IOQ,
+            get_kv_node = KVN,
+            get_kp_node = KPN,
+            docs_read = Docs,
+            rows_read = Rows
+        } = R
+    ) when
+        DbName =:= DbName1 andalso
+            ((IOQ >= Threshold) or (KVN >= Threshold) or (KPN >= Threshold) or (Docs >= Threshold) or
+                (Rows >= Threshold))
+    ->
+        R
+    end).
+
+-spec matcher_on_docs_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_read=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_read = DocsRead} = R) when DocsRead >= Threshold -> R end).
+
+-spec matcher_on_docs_written(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_docs_written(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %%ets:fun2ms(fun(#rctx{type=#coordinator{}, docs_written=DocsRead} = R) when DocsRead >= Threshold -> R end).
+    ets:fun2ms(fun(#rctx{docs_written = DocsWritten} = R) when DocsWritten >= Threshold -> R end).
+
+-spec matcher_on_rows_read(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_rows_read(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{rows_read = RowsRead} = R) when RowsRead >= Threshold -> R end).
+
+-spec matcher_on_nonce(Nonce :: nonce()) -> ets:match_spec().
+matcher_on_nonce(Nonce) ->
+    ets:fun2ms(fun(#rctx{nonce = Nonce1} = R) when Nonce =:= Nonce1 -> R end).
+
+-spec matcher_on_changes_processed(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_changes_processed(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% HACK: because we overload the use of #rctx.rows_read for
+    %% changes_processed, we must specify a direct match against a changes
+    %% context. Fow now, just match on #coordinator's
+    ets:fun2ms(
+        fun(
+            #rctx{
+                type = #coordinator{mod = chttpd_db, func = handle_changes_req},
+                rows_read = Processed,
+                changes_returned = Returned
+            } = R
+        ) when (Processed - Returned) >= Threshold ->
+            R
+        end
+    ).
+
+%% Matcher on requests taking longer than Threshold milliseconds
+-spec matcher_on_long_reqs(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_long_reqs(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    %% Threshold is in milliseconds, but we track erlang:monotonic_time/0
+    %% which is in native format, a machine dependent internal representation
+    %% but with the guarantee of monotonically increasing. Note that this number
+    %% is integer(), _not_ positive_integer(), so we must use abs/2 to get a
+    %% positive time delta; notably, abs/2 is valid match_spec guard.
+    %%
+    %% Time warps and is relative and is complicated, so here's an example of
+    %% converting 10000 milliseconds into a native time format and back, then
+    %% using csrt_util:tnow/0 to accurately measure sleeping for 10000 ms.
+    %%
+    %% (node1@127.0.0.1)5> erlang:convert_time_unit(10000, millisecond, native).
+    %% 10000000000
+    %% (node1@127.0.0.1)6> erlang:convert_time_unit(10000000000, native, millisecond).
+    %% 10000
+    %% (node1@127.0.0.1)7> T0 = csrt_util:tnow(), timer:sleep(10000), T1 = csrt_util:tnow(),
+    %%     erlang:convert_time_unit(abs(T1 - T0), native, millisecond).
+    %% 10000
+
+    Unit = millisecond,
+    NativeThreshold = erlang:convert_time_unit(Threshold, Unit, native),
+    ets:fun2ms(
+        fun(
+            #rctx{
+                started_at = Started,
+                updated_at = Updated
+            } = R
+        ) when abs(Updated - Started) >= NativeThreshold ->
+            R
+        end
+    ).
+
+-spec matcher_on_ioq_calls(Threshold :: pos_integer()) -> ets:match_spec().
+matcher_on_ioq_calls(Threshold) when
+    is_integer(Threshold) andalso Threshold > 0
+->
+    ets:fun2ms(fun(#rctx{ioq_calls = IOQCalls} = R) when IOQCalls >= Threshold -> R end).
+
+%%-spec matcher_for_rctx_field(Field :: rctx_field()) -> ets:match_spec().
+%%matcher_for_rctx_field() ->
+%%    #{size := Size0, fields := Fields} = csrt_entry:record_info(),
+%%    %% Subtract 1 as record_info size includes tuple record name
+%%    %% erlang:list_to_tuple([rctx | lists:duplicate(maps:get(size, csrt_entry:record_info()), '_')])
+%%    Size = Size - 1,

Review Comment:
   I've deleted this now, but this was the start to my replacement to the `ets:fun2ms` parse transform, as the logic there will dynamically setup the matcher string on the appropriate field index, but I didn't have a chance to complete it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221121077


##########
src/couch_stats/CSRT.md:
##########
@@ -0,0 +1,893 @@
+# Couch Stats Resource Tracker (CSRT)
+
+CSRT (Couch Stats Resource Tracker) is a real time stats tracking system that
+tracks the quantity of resources induced at the process level in a live
+queryable manner that also generates process lifetime reports containing
+statistics on the total resource load of a request, as a function of things like
+dbs/docs opened, view and changes rows read, changes returned vs processed,
+Javascript filter usage, duration, and more. This system is a paradigm shift in
+CouchDB visibility and introspection, allowing for expressive real time querying
+capabilities to introspect, understand, and aggregate CouchDB internal resource
+usage, as well as powerful filtering facilities for conditionally generating
+reports on "heavy usage" requests or "long/slow" requests. CSRT also extends
+`recon:proc_window` with `csrt:proc_window` allowing for the same style of
+battle hardened introspection with Recon's excellent `proc_window`, but with the
+sample window over any of the CSRT tracked CouchDB stats!
+
+CSRT does this by piggy-backing off of the existing metrics tracked by way of
+`couch_stats:increment_counter` at the time when the local process induces those
+metrics inc calls, and then CSRT updates an ets entry containing the context
+information for the local process, such that global aggregate queries can be
+performed against the ets table as well as the generation of the process
+resource usage reports at the conclusions of the process's lifecyle.The ability
+to do aggregate querying in realtime in addition to the process lifecycle
+reports for post facto analysis over time, is a cornerstone of CSRT that is the
+result of a series of iterations until a robust and scalable aproach was built.
+
+The real time querying is achieved by way of a global ets table with
+`read_concurrency`, `write_concurrency`, and `decentralized_counters` enabled.
+Great care was taken to ensure that _zero_ concurrent writes to the same key
+occure in this model, and this entire system is predicated on the fact that
+incremental updates to `ets:update_counters` provides *really* fast and
+efficient updates in an atomic and isolated fashion when coupled with
+decentralized counters and write concurrency. Each process that calls
+`couch_stats:increment_counter` tracks their local context in CSRT as well, with
+zero concurrent writes from any other processes. Outside of the context setup
+and teardown logic, _only_ operations to `ets:update_counter` are performed, one
+per process invocation of `couch_stats:increment_counter`, and one for
+coordinators to update worker deltas in a single batch, resulting in a 1:1 ratio
+of ets calls to real time stats updates for the primary workloads.
+
+The primary achievement of CSRT is the core framework iself for concurrent
+process local stats tracking and real time RPC delta accumulation in a scalable
+manner that allows for real time aggregate querying and process lifecycle
+reports. This took several versions to find a scalable and robust approach that
+induced minimal impact on maximum system throughput. Now that the framework is
+in place, it can be extended to track any further desired process local uses of
+`couch_stats:increment_counter`. That said, the currently selected set of stats
+to track was heavily influenced by the challenges in reotractively understanding
+the quantity of resources induced by a query like `/db/_changes?since=$SEQ`, or
+similarly, `/db/_find`.
+
+CSRT started as an extension of the Mango execution stats logic to `_changes`
+feeds to get proper visibility into quantity of docs read and filtered per
+changes request, but then the focus inverted with the realization that we should
+instead use the existing stats tracking mechanisms that have already been deemed
+critical information to track, which then also allows for the real time tracking
+and aggregate query capabilities. The Mango execution stats can be ported into
+CSRT itself and just become one subset of the stats tracked as a whole, and
+similarly, any additional desired stats tracking can be easily added and will
+be picked up in the RPC deltas and process lifetime reports.
+
+# CSRT Config Keys
+
+## -define(CSRT, "csrt").
+
+> config:get("csrt").
+
+Primary CSRT config namespace: contains core settings for enabling different
+layers of functionality in CSRT, along with global config settings for limiting
+data volume generation.
+
+## -define(CSRT_MATCHERS_ENABLED, "csrt_logger.matchers_enabled").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config toggles for enabling specific builtin logger matchers, see the dedicated
+section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+> config:get("csrt_logger.matchers_threshold").
+
+Config settings for defining the primary `Threshold` value of the builtin logger
+matchers, see the dedicated section below on `# CSRT Default Matchers`.
+
+## -define(CSRT_MATCHERS_DBNAMES, "csrt_logger.dbnames_io").
+
+> config:get("csrt_logger.matchers_enabled").
+
+Config section for setting `$db_name = $threshold` resulting in instantiating a
+"dbname_io" logger matcher for each `$db_name` that will generate a CSRT
+lifecycle report for any contexts that that induced more operations on _any_ one
+field of `ioq_calls|get_kv_node|get_kp_node|docs_read|rows_read` that is greater
+than `$threshold` and is on database `$db_name`.
+
+This is basically a simple matcher for finding heavy IO requests on a particular
+database, in a manner amenable to key/value pair specifications in this .ini
+file until a more sophisticated declarative model exists. In particular, it's
+not easy to sequentially generate matchspecs by way `ets:fun2ms/1`, and so an
+alternative mechanism for either dynamically assembling an `#rctx{}` to match
+against or generating the raw matchspecs themselves is warranted.
+
+## -define(CSRT_INIT_P, "csrt.init_p").
+
+> config:get("csrt.init_p").
+
+Config toggles for tracking counters on spawning of RPC `fabric_rpc` workers by
+way of `rexi_server:init_p`. This allows us to conditionally enable new metrics
+for the desired RPC operations in an expandable manner, without having to add
+new stats for every single potential RPC operation. These are for the individual
+metrics to track, the feature is enabled by way of the config toggle
+`config:get(?CSRT, "enable_init_p")`, and these configs can left alone for the
+most part until new operations are tracked.
+
+# CSRT Code Markers
+
+## -define(CSRT_ETS, csrt_server).
+
+This is the reference to the CSRT ets table, it's managed by `csrt_server` so
+that's where the name originates from.
+
+## -define(MATCHERS_KEY, {csrt_logger, all_csrt_matchers}).
+
+This marker is where the active matchers are written to in `persisten_term` for
+concurrently and parallelly and accessing the logger matchers in the CSRT
+tracker processes for lifecycle reporting.
+
+# CSRT Process Dictionary Markers
+
+## -define(PID_REF, {csrt, pid_ref}).
+
+This marker is for the core storing the core `PidRef` identifier. The key idea
+here is that a lifecycle is a context lifecycle is contained to within the given
+`PidRef`, meaning that a `Pid` can instantiate different CSRT lifecycles and
+pass those to different workers.
+
+This is specifically necessary for long running processes that need to handle
+many CSRT context lifecycles over the course of that individual process's
+lifecycle independent. In practice, this is immediately needed for the actual
+coordinator lifecycle tracking, as `chttpd` uses a worker pool of http request
+handlers that can be re-used, so we need a way to create a CSRT lifecycle
+corresponding to the given request currently being serviced. This is also
+intended to be used in other long running processes, like IOQ or `couch_js` pids
+such that we can track the specific context inducing the operations on the
+`couch_file` pid or indexer or replicator or whatever.
+
+Worker processes have a more clear cut lifecycle, but either style of process
+can be exit'ed in a manner that skips the ability to do cleanup operations, so
+additionally there's a dedicated tracker process spawned to monitor the process
+that induced the CSRT context such that we can do the dynamic logger matching
+directly in these tracker processes and also we can properly cleanup the ets
+entries even if the Pid crashes.
+
+## -define(TRACKER_PID, {csrt, tracker}).
+
+A handle to the spawned tracker process that does cleanup and logger matching
+reprots at the end of the process lifecycle. We store a reference to the tracker
+pid so that for explicit context destruction, like in `chttpd` workers after a
+request has been serviced, we can update stop the tracker and perform the
+expected cleanup directly.
+
+## -define(DELTA_TA, {csrt, delta_ta}).
+
+This stores our last delta snapshot to track progress since the last incremental
+streaming of stats back to the coordinator process. This will be updated after
+the next delta is made with the latest value. Eg this stores `T0` so we can do
+`T1 = get_resource()` `make_delta(T0, T1)` and then we save `T1` as the new `T0`
+for use in our next delta.
+
+## -define(LAST_UPDATED, {csrt, last_updated}).
+
+This stores the integer corresponding to the `erlang:monotonic_time()` value of
+the most recent `updated_at` value. Basically this lets us utilize a pdict
+value to be able to turn `update_at` tracking into an incremental operation that
+can be chained in the existing atomic `ets:update_counter` and
+`ets:update_element` calls.
+
+The issue being that our updates are of the form `+2 to ioq_calls for $pid_ref`,
+which ets does atomically in a guaranteed `atomic` and `isolated` manner. The
+strict use of the atomic operations for tracking these values is why this
+system works effeciently at scale. This means that we can increment counters on
+all of the stats counter fields in a batch, very quickly, but for tracking
+`updated_at` timestamps we'd need to either do an extra ets call to get the last
+`updated_at` value, or do an extra ets call to `ets:update_element` to set the
+`updated_at` value to `csrt_util:tnow()`. The core problem with this is that the
+batch inc operation is essentially the only write operation performed after the
+initial context setting of dbname/handler/etc; this means that we'd literally
+double the number of ets calls induced to track CSRT updates, just for tracking
+the `updated_at`. So instead, we rely on the fact that the local process
+corresponding to `$pid_ref` is the _only_ process doing updates so we know the
+last `updated_at` value will be the last time this process updated the data. So
+we track that value in the pdict and then take a delta between `tnow()` and
+`updated_at`, and then `updated_at` becomes a value we can sneak into the other
+integer counter updates we're already performing!
+
+# Primary Config Toggles
+
+# CSRT (?CSRT="csrt") Config Settings
+
+## config:get(?CSRT, "enable", false).
+
+Core enablement toggle for CSRT, defaults to false. Enabling this setting
+intiates local CSRT stats collection as well as shipping deltas in RPC
+responses to accumulate in the coordinator.
+
+This does _not_ trigger the new RPC spawn metrics, and it does not enable
+reporting for any of the rctx types.
+
+*NOTE*: you *MUST* have all nodes in the cluster running a CSRT aware CouchDB
+_before_ you enable it on any node, otherwise the old version nodes won't know
+how to handle the new RPC formats including an embedded Delta payload.
+
+## config:get(?CSRT, "enable_init_p", false).
+
+Enablement of tracking new metric counters for different `fabric_rpc` operations
+types to track spawn rates of RPC work induced across the cluster. There is
+corresponding config lookups into the `?CSRT_INIT_P` namespace for keys of the
+form: `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg `"fabric_rpc__open_doc"`
+for enabling the specific RPC endpoints.
+
+However, those individual settings can be ignored and this top level config
+toggle is what should be used in general, as the function specific config
+toggles predominantly exist to enable tracking a subet of total RPC operations
+in the cluster, and new endpoints can be added here.
+
+## config:get(?CSRT, "enable_reporting", false).
+
+This is the primary toggle for enabling CSRT process lifetime reports containing
+detailed information about the quantity of work induced by the given
+request/worker/etc. This is the top level toggle for enabling _any_ reporting,
+and there also exists `config:get(?CSRT, "enable_rpc_reporting", false).` to
+disable the reporting of any individual RPC workers, leaving the coordinator
+responsible of generating a report with the accumulated deltas.
+
+## config:get(?CSRT, "enable_rpc_reporting", false).
+
+This enables the possibility of RPC workers generating reports. They still need
+to hit the configured thresholds to induce a report, but this will generate CSRT
+process lifetime reports for individual RPC workers that trigger the configured
+logger thresholds. This allows for quantifying per node resource usage when
+desired, as otherwise the reports are at the http request level and don't
+provide per node stats.
+
+The key idea here is that having RPC level CSRT process lifetime reporting is
+incredibly useful, but can also generate large quantities of data. For example,
+a view query on a Q=64 database will stream results from 64 shard replicas,
+resulting in at least 64 RPC reports, plus any that might have been generated
+from RPC workers that "lost" the race for shard replica. This is very useful,
+but a lot of data given the verbose nature of funneling it through the RSyslog
+reports, however, the ability to write directly to something like ClickHouse or
+another columnar store would be great.
+
+Until there's an efficient storage mechanism to stream the results to, the
+rsyslog entries work great and are very practical, but care must be taken to
+not generate too much data for aggregate queries as they generate at least `Qx`
+more report than an individual report per http request from the coordinator.
+This setting exists as a way to either a) utilize the logger matcher configured
+thresholds to allow for _any_ rctx's to be recorded when they induce heavy
+operations, either Coordinator or RPC worker; or b) to _only_ log workloads at
+the coordinator level.
+
+NOTE: this setting exists because we lack an expressive enough config
+declaration to easily chain the matchspec constructions as `ets:fun2ms/1` is a
+special compile time parse transform macro that requires the fully definition to
+be specified directly, it cannot be iteractively constructed. That said, you
+_can_ register matchers through remsh with more specific and fine grained
+pattern matching, and a more expressive system for defining matchers are being
+explored.
+
+## config:get_boolean(?CSRT, "should_truncate_reports", true)
+
+Enables truncation of the CSRT process lifetime reports to not include any
+fields that are zero at the end of process lifetime, eg don't include
+`js_filter=0` in the report if the request did not induce Javascript filtering.
+
+This can be disabled if you really care about consistent fields in the report
+logs, but this is a log space saving mechanism, similar to disabling RPC
+reporting by default, as its a simple way to reduce overall volume
+
+## config:get(?CSRT, "randomize_testing", true).
+
+This is a `make eunit` only feature toggle that will induce randomness into the
+cluster's `csrt:is_enabled()` state, specifically to utilize the test suite to
+exercise edge case scenarios and failures when CSRT is only conditionally
+enabled, ensuring that it gracefuly and robustly handles errors without fallout
+to the underlying http clients.
+
+The idea here is to introduce randomness into whether CSRT is enabled across all
+the nodes to simulate clusters with heterogeneous CSRT enablement and also to
+ensure that CSRT works properly when toggled on/off wihout causing any
+unexpected fallout to the client requests.
+
+This is a config toggle specifically so that the actual CSRT tests can disable
+it for making accurate assertions about resource usage traacking, and is not
+intended to be used directly.
+
+## config:get_integer(?CSRT, "query_limit", ?QUERY_LIMIT)
+
+Limit the quantity of rows that can be loaded in an http query.
+
+# CSRT_INIT_P (?CSRT_INIT_P="csrt.init_p") Config Settings
+
+## config:get(?CSRT_INIT_P, ModFunName, false).
+
+These config toggles exist to conditionaly enable additional tracking of RPC
+endpoints of interest, but rather it's a way to selectively enable tracking for
+a subset of RPC operations, in a way we can extend later to add more. The
+`ModFunName` is of the form `atom_to_list(Mod) ++ "__" atom_to_list(Fun)`, eg
+`"fabric_rpc__open_doc"`, and right now, only exists for `fabric_rpc` modules.
+
+NOTE: this is a bit awkward and isn't meant to be used directly, instead,
+utilize `config:set(?CSRT, "enable_init_p", "true").` to enable or disable these
+as a whole.
+
+The current set of operations, as copied in from `default.ini`
+
+```
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+```
+
+# CSRT Logger Matcher Enablement and Thresholds
+
+There are currently six builtin default loggers designed to make it easy to do
+filtering on heavy resource usage inducing and long running requests. These are
+designed as a simple baseline of useful matchers, declared in a manner amenable
+to `default.ini` based constructs. More expressive matcher declarations are
+being explored, and matchers of arbitrary complexity can be registered directly
+through remsh. The default matchers are all designed around an integer config
+Threshold that triggers on a specific field, eg docs read, or on a delta of
+fields for long requests and changes requests that process many rows but return
+few.
+
+The current default matchers are:
+
+  * docs_read: match all requests reading more than N docs
+  * rows_read: match all requests reading more than N rows
+  * docs_written: match all requests writing more than N docs
+  * long_reqs: match all requests lasting more than N milliseconds
+  * changes_processed: match all changes requests that returned at least N rows
+    less than was necessarily loaded to complete the request (eg find heavy
+    filtered changes requests reading many rows but returning few).
+  * ioq_calls: match all requests inducing more than N ioq_calls
+
+Each of the default matchers has an enablement setting in
+`config:get(?CSRT_MATCHERS_ENABLED, Name)` for toggling enablement of it, and a
+corresponding threshold value setting in `config:get(?CSRT_MATCHERS_THRESHOLD,
+Name)` that is an integer value corresponding to the specific nature of that
+matcher.
+
+## CSRT Logger Matcher Enablement (?CSRT_MATCHERS_ENABLED)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_enabled").
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_read", false)
+
+Enable the `docs_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `rows_read` filter in that a view with `?limit=1000`
+will read 1000 rows, but the same request with `?include_docs=true` will also
+induce an additional 1000 docs read.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "rows_read", false)
+
+Enable the `rows_read` builtin matcher, with a default `Threshold=1000`, such
+that any request that reads more than `Threshold` rows will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+This is different from the `docs_read` filter so that we can distinguish between
+heavy view requests with lots of rows or heavy requests with lots of docs.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "docs_written", false)
+
+Enable the `docs_written` builtin matcher, with a default `Threshold=500`, such
+that any request that writtens more than `Threshold` docs will generate a CSRT
+process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "ioq_calls", false)
+
+Enable the `ioq_calls` builtin matcher, with a default `Threshold=10000`, such
+that any request that induces more than `Threshold` IOQ calls will generate a
+CSRT process lifetime report with a summary of its resouce consumption.
+
+### config:get_boolean(?CSRT_MATCHERS_ENABLED, "long_reqs", false)
+
+Enable the `long_reqs` builtin matcher, with a default `Threshold=60000`, such
+that any request where the the last CSRT rctx `updated_at` timestamp is at least
+`Threshold` milliseconds grather than the `started_at timestamp` will generate a
+CSRT process lifetime report with a summary of its resource consumption.
+
+## CSRT Logger Matcher Threshold (?CSRT_MATCHERS_THRESHOLD)
+
+> -define(CSRT_MATCHERS_THRESHOLD, "csrt_logger.matchers_threshold").
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_read", 1000)
+
+Threshold for `docs_read` logger matcher, defaults to `1000` docs read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "rows_read", 1000)
+
+Threshold for `rows_read` logger matcher, defaults to `1000` rows read.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "docs_written", 500)
+
+Threshold for `docs_written` logger matcher, defaults to `500` docs written.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "ioq_calls", 10000)
+
+Threshold for `ioq_calls` logger matcher, defaults to `10000` IOQ calls made.
+
+### config:get_integer(?CSRT_MATCHERS_THRESHOLD, "long_reqs", 60000)
+
+Threshold for `long_reqs` logger matcher, defaults to `60000` milliseconds.
+
+# Core CSRT API
+
+The `csrt(.erl)` module is the primary entry point into CSRT, containing API
+functionality for tracking the lifecycle of processes, inducing metric tracking
+over that lifecycle, and also a variety of functions for aggregate querying.
+
+It's worth noting that the CSRT context tracking functions are specifically
+designed to not `throw` and be safe in the event of unexpected CSRT failures or
+edge cases. The aggregate query API has some callers that will actually throw,
+but aside from this core CSRT operations will not bubble up exceptions, and will
+either return the error value, or catch the error and move on rather than
+chaining further errors.
+
+## PidRef API
+
+These are functions are CRUD operations around creating and storing the CSRT
+`PidRef` handle.
+
+```
+-export([
+    destroy_pid_ref/0,
+    destroy_pid_ref/1,
+    create_pid_ref/0,
+    get_pid_ref/0,
+    get_pid_ref/1,
+    set_pid_ref/1
+]).
+```
+
+## Context Lifecycle API
+
+These are the CRUD functions for handling a CSRT context lifecycle, where a
+lifecycle context is created in a `chttpd` coordinator process by way of
+`csrt:create_coordinator_context/2`, or in `rexi_server:init_p` by way of
+`csrt:create_worker_context/3`. Additional functions are exposed for setting
+context specific info like username/dbname/handler. `get_resource` fetches the
+context being tracked corresponding to the given `PidRef`.
+
+```
+-export([
+    create_context/2,
+    create_coordinator_context/2,
+    create_worker_context/3,
+    destroy_context/0,
+    destroy_context/1,
+    get_resource/0,
+    get_resource/1,
+    set_context_dbname/1,
+    set_context_dbname/2,
+    set_context_handler_fun/1,
+    set_context_handler_fun/2,
+    set_context_username/1,
+    set_context_username/2
+]).
+```
+
+## Public API
+
+The "Public" or miscellaneous API for lack of a better name. These are various
+functions exposed for wider use and/or testing purposes.
+
+```
+-export([
+    clear_pdict_markers/0,
+    do_report/2,
+    is_enabled/0,
+    is_enabled_init_p/0,
+    maybe_report/2,
+    to_json/1
+]).
+```
+
+## Stats Collection API
+
+This is the stats collection API utilized by way of
+`couch_stats:increment_counter` to do local process tracking, and also in `rexi`
+to adding and extracting delta contexts and then accumulating those values.
+
+NOTE: `make_delta/0` is a "destructive" operation that will induce a new delta
+by way of the last local pdict's rctx delta snapshot, and then update to the
+most recent version. Two individual rctx snapshots for a PidRef can safely
+generate an actual delta by way of `csrt_util:rctx_delta/2`.
+
+```
+-export([
+    accumulate_delta/1,
+    add_delta/2,
+    docs_written/1,
+    extract_delta/1,
+    get_delta/0,
+    inc/1,
+    inc/2,
+    ioq_called/0,
+    js_filtered/1,
+    make_delta/0,
+    rctx_delta/2,
+    maybe_add_delta/1,
+    maybe_add_delta/2,
+    maybe_inc/2,
+    should_track_init_p/1
+]).
+```
+
+## TODO: RPC/QUERY DOCS
+
+```
+%% RPC API
+-export([
+    rpc/2,
+    call/1
+]).
+
+%% Aggregate Query API
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+    count_by/1,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+    group_by/2,
+    group_by/3,
+    query/1,
+    query/2,
+    query_matcher/1,
+    query_matcher/2,
+    sorted/1,
+    sorted_by/1,
+    sorted_by/2,
+    sorted_by/3
+]).
+```
+
+## Recon API Ports of https://github.com/ferd/recon/releases/tag/2.5.6
+
+This is a "port" of `recon:proc_window` to `csrt:proc_window`, allowing for
+`proc_window` style aggregations/sorting/filtering but with the stats fields
+collected by CSRT! This is also a direct port of `recon:proc_window` in that it
+utilizes the same underlying logic and effecient internal data structures as
+`recon:proc_window`, but rather only changes the Sample function:
+
+```erlang
+%% This is a recon:proc_window/3 [1] port with the same core logic but
+%% recon_lib:proc_attrs/1 replaced with pid_ref_attrs/1, and returning on
+%% pid_ref() rather than pid().
+%% [1] https://github.com/ferd/recon/blob/c2a76855be3a226a3148c0dfc21ce000b6186ef8/src/recon.erl#L268-L300
+-spec proc_window(AttrName, Num, Time) -> term() | throw(any()) when
+    AttrName :: rctx_field(), Num :: non_neg_integer(), Time :: pos_integer().
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    {First, Last} = recon_lib:sample(Time, Sample),
+    recon_lib:sublist_top_n_attrs(recon_lib:sliding_window(First, Last), Num).
+```
+
+In particular, our change is `Sample = fun() -> pid_ref_attrs(AttrName) end,`,
+and in fact, if recon upstream parameterized the option of `AttrName` or
+`SampleFunction`, this could be reimplemented as:
+
+```erlang
+%% csrt:proc_window
+proc_window(AttrName, Num, Time) ->
+    Sample = fun() -> pid_ref_attrs(AttrName) end,
+    recon:proc_window(Sample, Num, Time).
+```
+
+This implementation is being highlighted here because `recon:proc_window/3` is
+battle hardened and `recon_lib:sliding_window` uses an effecient internal data
+structure for storing the two samples that has been proven to work in production
+systems with millions of active processes, so swapping the `Sample` function
+with a CSRT version allows us to utilize the production grade recon
+functionality, but extended out to the particular CouchDB statistics we're
+esepecially interested in.
+
+And on a fun note: any further stats tracking fields added to CSRT tracking will
+automatically work with this too.
+
+
+```
+-export([
+    pid_ref_attrs/1,
+    pid_ref_matchspec/1,
+    proc_window/3
+]).
+```
+
+<hr />
+
+# Core types and Maybe types
+
+Before we look at the `#rctx{}` record fields, lets examine the core datatypes
+defined by CSRT for use in Dialyzer typespecs. There are more, but these are the
+essentials and demonstrate the "maybe" typespec approach utilized in CSRT.
+
+Let's say we have a `-type foo() :: #foo{}` and `-type maybe_foo() :: foo() |
+undefined`, we then can construct functions of the form `-spec get_foo(id()) ->
+maybe_foo()` and then we can use Dialyzer to statically assert all callers of
+`get_foo/1` handle the `maybe_foo()` data type rather than just `foo()` and
+ensure that all subsequent callers do as well.
+
+This approach of `-spec maybe_<Type> :: <Type> | undefined` is utilized
+throughout CSRT and has greatly aided in the development, refactoring, and
+static analysis of this system. Here's a useful snippet for running Dialyzer
+while hacking on CSRT:
+
+> make && time make dialyze apps=couch_stats
+
+```erlang
+-type pid_ref() :: {pid(), reference()}.
+-type maybe_pid_ref() :: pid_ref() | undefined.
+
+-type coordinator_rctx() :: #rctx{type :: coordinator()}.
+-type rpc_worker_rctx() :: #rctx{type :: rpc_worker()}.
+-type rctx() :: #rctx{} | coordinator_rctx() | rpc_worker_rctx().
+-type rctxs() :: [#rctx{}] | [].
+-type maybe_rctx() :: rctx() | undefined.
+```
+
+Above we have the core `pid_ref()` data type, which is just a tuple with a
+`pid()` and a `reference()`, and naturally, `maybe_pid_ref()` handles the
+optional presence of a `pid_ref()`, allowing for our APIs like
+`csrt:get_resource(maybe_pidref())` to handle ambiguity of the presence of a
+`pid_ref()`.
+
+We define our core `rctx()` data type as an empty `#rctx{}`, or the more
+specific `coordinator_rctx()` or `rpc_worker_rctx()` such that we can be
+specific about the `rctx()` type in functions that need to distinguish. And then
+as expected, we have the notion of `maybe_rctx()`.
+
+# #rctx{}
+
+This is the core data structure utilized to track a CSRT context for a
+coordinator or rpc_worker process, represented by the `#rctx{}` record, and
+stored in the `?CSRT_ETS` table keyed on `{keypos, #rctx.pid_ref}`.
+
+The Metadata fields store labeling data for the given process being tracked,
+such as started_at and updated_at timings, the primary `pid_ref` id key, the
+type of the process context, and some additional information like username,
+dbname, and the nonce of the coordinator request.
+
+The Stats Counters fields are `non_neg_integer()` monotonically increasing
+counters corresponding to the `couch_stats` metrics counters we're interested in
+tracking at a process level cardinality. The use of these purely integer counter
+fields represented by a record represented in an ets table is the cornerstone of
+CSRT and why its able to operate at high throughput and high concurrency, as
+`ets:update_counter/{3,4}` take increment operations to be performed atomically
+and in isolation, in a manner in which does not require fetching and loading the
+data directly. We then take care to batch the accumulation of delta updates into
+a single `update_counter` call and even sneak in the `updated_at` tracking as a
+integer counter update without inducing an extra ets call.
+
+NOTE: the typespec's for these fields include `'_'` atoms as possible types as
+that is the matchspec wildcard any of the fields can be set to when using an
+existing `#rctx{}` record to search with.
+
+
+```erlang
+-record(rctx, {
+    %% Metadata
+    started_at = csrt_util:tnow() :: integer() | '_',
+    %% NOTE: updated_at must be after started_at to preserve time congruity
+    updated_at = csrt_util:tnow() :: integer() | '_',
+    pid_ref :: maybe_pid_ref() | {'_', '_'} | '_',
+    nonce :: nonce() | undefined | '_',
+    type :: rctx_type() | undefined | '_',
+    dbname :: dbname() | undefined | '_',
+    username :: username() | undefined | '_',
+
+    %% Stats Counters
+    db_open = 0 :: non_neg_integer() | '_',
+    docs_read = 0 :: non_neg_integer() | '_',
+    docs_written = 0 :: non_neg_integer() | '_',
+    rows_read = 0 :: non_neg_integer() | '_',
+    changes_returned = 0 :: non_neg_integer() | '_',
+    ioq_calls = 0 :: non_neg_integer() | '_',
+    js_filter = 0 :: non_neg_integer() | '_',
+    js_filtered_docs = 0 :: non_neg_integer() | '_',
+    get_kv_node = 0 :: non_neg_integer() | '_',
+    get_kp_node = 0 :: non_neg_integer() | '_'
+    %% "Example to extend CSRT"
+    %%write_kv_node = 0 :: non_neg_integer() | '_',
+    %%write_kp_node = 0 :: non_neg_integer() | '_'
+}).
+```
+
+## Metadata
+
+We use `csrt_util:tnow()` for time tracking, which is a `native` format
+`erlang:monotonic_time()` integer, which, noteably, _can_ be and is often a
+negative value. You must either take a delta or convert the time to get into a
+useable format, as one might suspect by the use of `native`.
+
+We make use of `erlang:mononotic_time/0` as per the recommendation in
+https://www.erlang.org/doc/apps/erts/time_correction.html#how-to-work-with-the-new-api
+for the suggested way to `Measure Elasped Time`, as quoted:
+
+```
+Take time stamps with erlang:monotonic_time/0 and calculate the time difference
+using ordinary subtraction. The result is in native time unit. If you want to
+convert the result to another time unit, you can use erlang:convert_time_unit/3.
+
+An easier way to do this is to use erlang:monotonic_time/1 with the desired time
+unit. However, you can then lose accuracy and precision.
+```

Review Comment:
   I added this documentation specifically because time in Erlang is confusing and my use of Erlang's time semantics are specific and intentional, so it's important that this be documented. There's clearly confusion on time aspects, as I know from implementing it and also from your comments and others on this PR and the previous ones.
   
   The linked documentation provides specific callouts of when and when not to use the different time components, so I want that to be directly recognizable in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221066712


##########
rel/overlay/etc/default.ini:
##########
@@ -1150,3 +1150,63 @@ url = {{nouveau_url}}
 ;mem3_shards = true
 ;nouveau_index_manager = true
 ;dreyfus_index_manager = true
+
+; Couch Stats Resource Tracker (CSRT)
+[csrt]
+enable = true
+enable_init_p = true
+enable_reporting = true
+;enable = false
+;enable_init_p = false
+;enable_reporting = false
+;enable_rpc_reporting = false
+
+; Truncate reports to not include zero values for counter fields. This is a
+; simple way to save space and should be left enabled unless you need a fixed
+; output structure in the process lifecycle reports.
+;should_truncate_reports = true
+
+; Limit queries to a maxinum number of rows
+;query_limit = 10000
+
+; CSRT Rexi Server RPC Worker Spawn Tracking
+; Enable these to enable additional metrics for RPC worker spawn rates
+; measuring how often RPC workers are spawned by way of rexi_server:init_p.
+; Mod and Function are separated by double underscores.
+[csrt.init_p]
+fabric_rpc__all_docs = true
+fabric_rpc__changes = true
+fabric_rpc__get_all_security = true
+fabric_rpc__map_view = true
+fabric_rpc__open_doc = true
+fabric_rpc__open_shard = true
+fabric_rpc__reduce_view = true
+fabric_rpc__update_docs = true
+
+; CSRT dbname matchers
+; Given a dbname and a positive integer, this will enable an IO matcher
+; against the provided db for any requests that induce IO in quantities
+; greater than the provided threshold on any one of: ioq_calls, rows_read
+; docs_read, get_kp_node, get_kv_node, or changes_processed.
+[csrt_logger.dbnames_io]
+; For example:
+; foo = 100
+; _dbs = 123
+; _users = 234
+; foo/bar = 200
+
+[csrt_logger.matchers_enabled]
+;docs_read = false
+;rows_read = false
+;docs_written = false
+;long_reqs = false
+;changes_processed = false
+;ioq_calls = false
+
+[csrt_logger.matchers_threshold]

Review Comment:
   See the CSRT.md docs, these are for the default matchers, there's `[csrt_logger.matchers_enabled]` and `[csrt_logger.matchers_threshold]` so that then for each of the default matchers we can check if its enabled and also fetch its threshold value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "chewbranca (via GitHub)" <gi...@apache.org>.
chewbranca commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2221179610


##########
src/couch_stats/src/csrt_server.erl:
##########
@@ -0,0 +1,318 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(csrt_server).
+
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    init/1,
+    handle_call/3,
+    handle_cast/2
+]).
+
+-export([
+    create_pid_ref/0,
+    create_resource/1,
+    destroy_resource/1,
+    get_resource/1,
+    get_context_type/1,
+    inc/2,
+    inc/3,
+    match_resource/1,
+    new_context/2,
+    set_context_dbname/2,
+    set_context_handler_fun/2,
+    set_context_type/2,
+    set_context_username/2,
+    update_counter/3,
+    update_counter/4,
+    update_counters/2,
+    update_counters/3
+]).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("csrt.hrl").
+
+-record(st, {}).
+
+%%
+%% Public API
+%%
+
+-spec create_pid_ref() -> pid_ref().
+create_pid_ref() ->
+    {self(), make_ref()}.
+
+%%
+%%
+%% Context lifecycle API
+%%
+
+-spec new_context(Type :: rctx_type(), Nonce :: nonce()) -> rctx().
+new_context(Type, Nonce) ->
+    #rctx{
+        nonce = Nonce,
+        pid_ref = create_pid_ref(),
+        type = Type
+    }.
+
+-spec set_context_dbname(DbName, PidRef) -> boolean() when
+    DbName :: dbname(), PidRef :: maybe_pid_ref().
+set_context_dbname(_, undefined) ->
+    false;
+set_context_dbname(DbName, PidRef) ->
+    update_element(PidRef, [{#rctx.dbname, DbName}]).
+
+-spec set_context_handler_fun({Mod, Func}, PidRef) -> boolean() when
+    Mod :: atom(), Func :: atom(), PidRef :: maybe_pid_ref().
+set_context_handler_fun(_, undefined) ->
+    false;
+set_context_handler_fun({Mod, Func}, PidRef) ->
+    case get_resource(PidRef) of
+        undefined ->
+            false;
+        #rctx{} = Rctx ->
+            %% TODO: #coordinator{} assumption needs to adapt for other types
+            case csrt_server:get_context_type(Rctx) of
+                #coordinator{} = Coordinator0 ->
+                    Coordinator = Coordinator0#coordinator{mod = Mod, func = Func},
+                    set_context_type(Coordinator, PidRef);
+                _ ->
+                    false
+            end
+    end.
+
+-spec set_context_username(UserName, PidRef) -> boolean() when
+    UserName :: username(), PidRef :: maybe_pid_ref().
+set_context_username(_, undefined) ->
+    false;
+set_context_username(UserName, PidRef) ->
+    update_element(PidRef, [{#rctx.username, UserName}]).
+
+-spec get_context_type(Rctx :: rctx()) -> rctx_type().
+get_context_type(#rctx{type = Type}) ->
+    Type.
+
+-spec set_context_type(Type, PidRef) -> boolean() when
+    Type :: rctx_type(), PidRef :: maybe_pid_ref().
+set_context_type(Type, PidRef) ->
+    update_element(PidRef, [{#rctx.type, Type}]).
+
+-spec create_resource(Rctx :: rctx()) -> boolean().
+create_resource(#rctx{} = Rctx) ->
+    (catch ets:insert(?CSRT_ETS, Rctx)) == true.
+
+-spec destroy_resource(PidRef :: maybe_pid_ref()) -> boolean().
+destroy_resource(undefined) ->
+    false;
+destroy_resource({_, _} = PidRef) ->
+    (catch ets:delete(?CSRT_ETS, PidRef)) == true.
+
+-spec get_resource(PidRef :: maybe_pid_ref()) -> maybe_rctx().
+get_resource(undefined) ->
+    undefined;
+get_resource(PidRef) ->
+    try ets:lookup(?CSRT_ETS, PidRef) of
+        [#rctx{} = Rctx] ->
+            Rctx;
+        [] ->
+            undefined
+    catch
+        _:_ ->
+            undefined
+    end.
+
+-spec match_resource(Rctx :: maybe_rctx()) -> [] | [rctx()].
+match_resource(undefined) ->
+    [];
+match_resource(#rctx{} = Rctx) ->
+    try
+        ets:match_object(?CSRT_ETS, Rctx)
+    catch
+        _:_ ->
+            []
+    end.
+
+%% Is this a valid #rctx{} field for inducing ets:update_counter upon?
+-spec is_rctx_stat_field(Field :: rctx_field() | atom()) -> boolean().
+is_rctx_stat_field(Field) ->
+    maps:is_key(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% Get the #rctx{} field record index of the corresponding stat counter field
+-spec get_rctx_stat_field(Field :: rctx_field()) ->
+    non_neg_integer()
+    | throw({badkey, Key :: any()}).
+get_rctx_stat_field(Field) ->
+    maps:get(Field, ?STAT_KEYS_TO_FIELDS).
+
+%% This provides a base set of updates to include along with any other #rctx{}
+%% updates. Specifically, this provides a way to automatically track and
+%% increment the #rctx.updated_at field without having to do ets:lookup to find
+%% the last updated_at time, or having to do ets:update_element to set a
+%% specific updated_at. We trade a pdict marker to keep inc operations as only
+%% a singular ets call while sneaking in updated_at.
+%% Calling csrt_util:put_updated_at/1 within this function is not the cleanest,
+%% but it allows us to encapsulate the automatic updated_at inclusion into the
+%% ?MODULE:update_counter(s)/3-4 arity call-through while still allowing the
+%% 4-arity version to be exposed to pass an empty base updates list. Isolating
+%% this logic means the final arity functions operate independently of any
+%% local pdict values.
+-spec make_base_counter_updates() -> counter_updates_list().
+make_base_counter_updates() ->
+    case csrt_util:get_updated_at() of
+        undefined ->
+            [];
+        LastUpdated ->
+            Now = csrt_util:tnow(),
+            csrt_util:put_updated_at(Now),
+            UpdatedInc = csrt_util:make_dt(LastUpdated, Now, native),
+            [{#rctx.updated_at, UpdatedInc}]
+    end.
+
+-spec update_counter(PidRef, Field, Count) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer().
+update_counter(undefined, _Field, _Count) ->
+    0;
+update_counter(_PidRef, _Field, 0) ->
+    0;
+update_counter(PidRef, Field, Count) ->
+    %% Only call make_base_counter_updates() if PidRef, Field, Count all valid
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, Count, make_base_counter_updates());
+        false ->
+            0
+    end.
+
+-spec update_counter(PidRef, Field, Count, BaseUpdates) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    Count :: non_neg_integer(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counter(undefined, _Field, _Count, _BaseUpdates) ->
+    0;
+update_counter({_Pid, _Ref} = PidRef, Field, Count, BaseUpdates) when Count >= 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            Updates = [{get_rctx_stat_field(Field), Count} | BaseUpdates],
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef})
+            catch
+                _:_ ->
+                    0
+            end;
+        false ->
+            0
+    end.
+
+-spec update_counters(PidRef, Delta) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta().
+update_counters(undefined, _Delta) ->
+    false;
+update_counters(PidRef, Delta) when is_map(Delta) ->
+    update_counters(PidRef, Delta, make_base_counter_updates()).
+
+-spec update_counters(PidRef, Delta, BaseUpdates) -> boolean() when
+    PidRef :: maybe_pid_ref(),
+    Delta :: delta(),
+    BaseUpdates :: [] | [{rctx_field(), integer()}].
+update_counters(undefined, _Delta, _BaseUpdates) ->
+    false;
+update_counters({_Pid, _Ref} = PidRef, Delta, BaseUpdates) when is_map(Delta) ->
+    Updates = maps:fold(
+        fun(Field, Count, Acc) ->
+            case is_rctx_stat_field(Field) of
+                true ->
+                    [{get_rctx_stat_field(Field), Count} | Acc];
+                false ->
+                    %% This skips entries that are not is_rctx_stat_field's
+                    %% Another approach would be:
+                    %% lists:all(lists:map(fun is_rctx_stat_field/1, maps:keys(Delta)))
+                    %% But that's a lot of looping for not even acumulating the update.
+                    %% Need to drop Delta.dt either way as it's not an rctx_field
+                    Acc
+            end
+        end,
+        BaseUpdates,
+        Delta
+    ),
+
+    case Updates of
+        [] ->
+            false;
+        _ ->
+            try
+                ets:update_counter(?CSRT_ETS, PidRef, Updates, #rctx{pid_ref = PidRef}),
+                true
+            catch
+                _:_ ->
+                    false
+            end
+    end.
+
+-spec inc(PidRef :: maybe_pid_ref(), Field :: rctx_field()) -> non_neg_integer().
+inc(PidRef, Field) ->
+    inc(PidRef, Field, 1).
+
+-spec inc(PidRef, Field, N) -> non_neg_integer() when
+    PidRef :: maybe_pid_ref(),
+    Field :: rctx_field(),
+    N :: non_neg_integer().
+inc(undefined, _Field, _) ->
+    0;
+inc(_PidRef, _Field, 0) ->
+    0;
+inc({_Pid, _Ref} = PidRef, Field, N) when is_integer(N) andalso N > 0 ->
+    case is_rctx_stat_field(Field) of
+        true ->
+            update_counter(PidRef, Field, N);
+        false ->
+            0
+    end.
+
+%%
+%% gen_server callbacks
+%%
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+init([]) ->
+    ets:new(?CSRT_ETS, [
+        named_table,
+        public,
+        {write_concurrency, auto},
+        {read_concurrency, true},
+        {keypos, #rctx.pid_ref}
+    ]),
+    {ok, #st{}}.
+
+handle_call(_, _From, State) ->
+    {reply, ok, State}.
+
+handle_cast(_Msg, State) ->
+    {noreply, State, 0}.

Review Comment:
   Oops no, that's just the generate a default timeout callback to the gen_server; leftover from previous experiments to trigger processing when the message queue is emptied. Removed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349759708


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   I would even skip the whole update_topK function and the record and just use the topk from above as is. There is no need to keeping track of size as gb data structure does that already, it's fine adding and removing an item in the same update, it's a log k  kind of a deal and k should already be a small constant anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349786080


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   > I would even skip the whole update_topK function and the record and just use the topk from above as is. 
   
   The `update_topK` skips the update when `Value < Min`, we would have many small values so we could safe a lot of updates.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "iilyak (via GitHub)" <gi...@apache.org>.
iilyak commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2352517704


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   Addressed in https://github.com/apache/couchdb/pull/5602/commits/27d4123d4313929727c108fa07b681f4eba9ba55



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349759708


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   I would even skip the whole update_topK and the record and just use the topk from above as is. There is no need to keeping track of size as gb data structure does that already, it's fine adding and removing an item in the same update, it's a log k  kind of a deal and log k should already be a small constant anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Couch stats resource tracker v3 rebase main [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5602:
URL: https://github.com/apache/couchdb/pull/5602#discussion_r2349885761


##########
src/couch_srt/src/couch_srt_query.erl:
##########
@@ -0,0 +1,985 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_srt_query).
+
+-feature(maybe_expr, enable).
+
+-include_lib("stdlib/include/ms_transform.hrl").
+-include_lib("couch_srt.hrl").
+
+%% aggregate query api
+-export([
+    active/0,
+    active/1,
+    active_coordinators/0,
+    active_coordinators/1,
+    active_workers/0,
+    active_workers/1,
+
+    all/0,
+    find_by_nonce/1,
+    find_by_pid/1,
+    find_by_pidref/1,
+    find_workers_by_pidref/1,
+
+    query_matcher/1,
+    query_matcher/2,
+    query_matcher_rows/1,
+    query_matcher_rows/2,
+
+    query/1,
+    from/1,
+    group_by/1,
+    group_by/2,
+    sort_by/1,
+    sort_by/2,
+    count_by/1,
+    options/1,
+    unlimited/0,
+    with_limit/1,
+
+    run/1,
+    unsafe_run/1
+]).
+
+-export_type([
+    query/0,
+    query_expression/0,
+    query_option/0
+]).
+
+-type aggregation_keys_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type value_key_fun() :: fun((Ele :: #rctx{}) -> aggregation_values() | aggregation_value()).
+-type count_key_fun() :: fun((A :: pos_integer(), B :: pos_integer()) -> pos_integer()).
+
+-record(selector, {
+    aggregation_keys = undefined ::
+        rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        rctx_field()
+        | undefined
+}).
+
+-record(unsafe_selector, {
+    aggregation_keys = undefined ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()]
+        | undefined,
+    value_key = undefined ::
+        value_key_fun()
+        | rctx_field()
+        | undefined
+}).
+
+-record(query_options, {
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-type aggregation() :: group_by | sort_by | count_by.
+
+-record(query, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    selector = undefined :: #selector{} | #unsafe_selector{} | undefined,
+    limit = undefined :: pos_integer() | unlimited | undefined,
+    aggregation = undefined :: aggregation() | undefined,
+    is_safe = true :: boolean()
+}).
+
+-record(from, {
+    matcher = undefined :: matcher_name() | all | undefined,
+    is_safe = undefined :: boolean() | undefined
+}).
+
+-opaque query() :: #query{}.
+-opaque query_expression() ::
+    #from{}
+    | #query_options{}
+    | #selector{}
+    | #unsafe_selector{}
+    | query_option()
+    | {aggregation(), #selector{}}
+    | {aggregation(), #unsafe_selector{}}.
+-opaque query_option() ::
+    {limit, pos_integer() | unlimited | undefined}.
+
+%%
+%% Aggregate query API
+%%
+
+active() ->
+    active_int(all).
+
+active_coordinators() ->
+    active_int(coordinators).
+
+active_workers() ->
+    active_int(workers).
+
+%% active_json() or active(json)?
+active(json) ->
+    to_json_list(active_int(all)).
+
+active_coordinators(json) ->
+    to_json_list(active_int(coordinators)).
+
+active_workers(json) ->
+    to_json_list(active_int(workers)).
+
+active_int(coordinators) ->
+    select_by_type(coordinators);
+active_int(workers) ->
+    select_by_type(workers);
+active_int(all) ->
+    select_by_type(all).
+
+select_by_type(coordinators) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #coordinator{}} = R) -> R end));
+select_by_type(workers) ->
+    ets:select(?CSRT_ETS, ets:fun2ms(fun(#rctx{type = #rpc_worker{}} = R) -> R end));
+select_by_type(all) ->
+    ets:tab2list(?CSRT_ETS).
+
+find_by_nonce(Nonce) ->
+    couch_srt_server:match_resource(#rctx{nonce = Nonce}).
+
+find_by_pid(Pid) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = {Pid, '_'}}).
+
+find_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{pid_ref = PidRef}).
+
+find_workers_by_pidref(PidRef) ->
+    couch_srt_server:match_resource(#rctx{type = #rpc_worker{from = PidRef}}).
+
+curry_field(Field) ->
+    fun(Ele) -> couch_srt_entry:value(Field, Ele) end.
+
+-spec group_by(Matcher, KeyFun, ValFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field().
+group_by(Matcher, KeyFun, ValFun) ->
+    AggFun = fun erlang:'+'/2,
+    group_by(Matcher, KeyFun, ValFun, AggFun).
+
+-spec group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun().
+group_by(Matcher, KeyFun, ValFun, AggFun) ->
+    group_by(Matcher, KeyFun, ValFun, AggFun, query_cardinality_limit()).
+
+-spec all() ->
+    matcher().
+
+all() ->
+    Spec = ets:fun2ms(fun(#rctx{} = R) -> R end),
+    {Spec, ets:match_spec_compile(Spec)}.
+
+%% eg: group_by(all(), username, docs_read).
+%% eg: ^^ or: group_by(all(), [username, docs_read], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], docs_read).
+%% eg: group_by(all(), [username, dbname, js_filter], ioq_calls).
+%% eg: group_by(all(), [username, dbname, js_filter], get_kv_node).
+-spec group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {ok, aggregation_result()} | {limit, aggregation_result()}
+when
+    Matcher :: matcher(),
+    KeyFun ::
+        aggregation_keys_fun()
+        | rctx_field()
+        | [rctx_field()],
+    ValFun ::
+        value_key_fun()
+        | rctx_field(),
+    AggFun ::
+        count_key_fun(),
+    Limit :: pos_integer().
+
+group_by(Matcher, KeyL, ValFun, AggFun, Limit) when is_list(KeyL) ->
+    KeyFun = fun(Ele) -> list_to_tuple([couch_srt_entry:value(Key, Ele) || Key <- KeyL]) end,
+    group_by(Matcher, KeyFun, ValFun, AggFun, Limit);
+group_by(Matcher, Key, ValFun, AggFun, Limit) when is_atom(Key) ->
+    group_by(Matcher, curry_field(Key), ValFun, AggFun, Limit);
+group_by(Matcher, KeyFun, Val, AggFun, Limit) when is_atom(Val) ->
+    group_by(Matcher, KeyFun, curry_field(Val), AggFun, Limit);
+group_by(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    %% This is a space versus speed tradeoff. Both query modes only filter
+    %% through the table until `Limit` rows have been returned and both will
+    %% utilize the compiled match_specs to do the testing, but
+    %% `group_by_fold/5` will sequentially copy in every row and test it
+    %% locally against the compiled match_spec using `ets:foldl/3`, whereas
+    %% `group_by_select/5` does the filtering internally in the ETS NIF, by way
+    %% of passing the uncompiled match_spec to `ets:select/3` to
+    %% `ets:select/3`. The tradeoff here is that `ets:select` will copy `Limit`
+    %% full `#rctx{}` records into this caller process, which we then aggregate
+    %% over, as opposed to `ets:foldl` only sequentially loading a singular
+    %% `#rctx{}` and extracting the relevant field value to aggregate on.
+    %%
+    %% The use of `query_by_fold` should only be needed if `Limit` is
+    %% drastically increased, and even then, the efficiencies gained here with
+    %% `query_by_fold` are shortlived until we can encode the values needed by
+    %% `ValFun` into the match_spec return fields, at which point it becomes
+    %% strictly worse.
+    %%
+    %% NOTE: This discrepancy of `ets:match_spec_run` taking a `match_spec()`
+    %% vs `ets:select` taking a `comp_match_spec()` is why our CSRT `matcher()`
+    %% type_spec funnels around both versions instead of just reference to the
+    %% compiled spec stored by ETS internally.
+    case config:get_boolean(?CSRT, "use_query_fold", false) of
+        true ->
+            group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit);
+        false ->
+            group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit)
+    end.
+
+group_by_fold(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    FoldFun = fun(Ele, Acc) ->
+        case maps:size(Acc) =< Limit of
+            true ->
+                case ets_match(Ele, Matcher) of
+                    true ->
+                        Key = KeyFun(Ele),
+                        Val = ValFun(Ele),
+                        CurrVal = maps:get(Key, Acc, 0),
+                        case AggFun(CurrVal, Val) of
+                            0 ->
+                                Acc;
+                            NewVal ->
+                                maps:put(Key, NewVal, Acc)
+                        end;
+                    false ->
+                        Acc
+                end;
+            false ->
+                throw({limit, Acc})
+        end
+    end,
+    try
+        {ok, ets:foldl(FoldFun, #{}, ?CSRT_ETS)}
+    catch
+        throw:{limit, Acc} ->
+            {limit, Acc}
+    end.
+
+ets_match(Ele, {_, CMS}) ->
+    ets:match_spec_run([Ele], CMS) =/= [].
+
+group_by_select(Matcher, KeyFun, ValFun, AggFun, Limit) ->
+    {Status, Rctxs} = group_by_select_rows(Matcher, Limit),
+    %% If we hit `Status=limit` rows, still aggregate over what we found
+    Aggregated = lists:foldl(
+        fun(Rctx, Acc) ->
+            Key = KeyFun(Rctx),
+            Val = ValFun(Rctx),
+            CurrVal = maps:get(Key, Acc, 0),
+            case AggFun(CurrVal, Val) of
+                0 ->
+                    Acc;
+                NewVal ->
+                    maps:put(Key, NewVal, Acc)
+            end
+        end,
+        #{},
+        Rctxs
+    ),
+    {Status, Aggregated}.
+
+group_by_select_rows(Matcher, Limit) ->
+    try
+        %% Use `ets:select/3` as this does the ets fold internally in a space
+        %% efficient way that is still faster than the sequential traversal
+        %% through the table. See the `ets:select/3` documentation for more
+        %% info. We also use `ets:select/3` to pass the limit along, which
+        %% results in ets effeciently traversing rows until `Limit` rows have
+        %% been accumulated and returned.
+        %% ets:select/* takes match_spec(), not  comp_match_spec()
+        {MSpec, _CMSpec} = Matcher,
+        case ets:select(?CSRT_ETS, MSpec, Limit) of
+            %% Technically the {Rctxs, `continuation()`} here is an `opaque()`
+            %% type, but we assume `'$end_of_table'` is a reasonable indication
+            %% of no more rows. However, we fallback to checking the quantity
+            %% returned in case this is ever no longer true.
+            {Rctxs, '$end_of_table'} ->
+                {ok, Rctxs};
+            {Rctxs, _Continuation} ->
+                %% Continuation is opaque, and there's no `is_more_rows` API to
+                %% check to see if we actually limit the table Limit or we hit
+                %% the edge case where exactly `Limit` rows were found.  The
+                %% continuation can be passed back to `ets:select/1` to see if
+                %% explicity returns `'$end_of_table'`, but if it did hit the
+                %% `Limit`, we now wastefully fetch the next chunk of rows, so
+                %% instead for now we assume that when the length of rows
+                %% equals `Limit` that we hit the cap.  Note that this is only
+                %% relevant because the API returning `'$end_of_table'` is not
+                %% formally specified, but in theory this clause should not be
+                %% hit.
+                case length(Rctxs) >= Limit of
+                    true ->
+                        {limit, Rctxs};
+                    false ->
+                        {ok, Rctxs}
+                end;
+            %% Handle '$end_of_table'
+            _ ->
+                {ok, []}
+        end
+    catch
+        _:_ ->
+            {ok, []}
+    end.
+
+%%
+%% Auxiliary functions to calculate topK
+%%
+
+-record(topK, {
+    % we store ordered elements in ascending order
+    seq = [] :: list({aggregation_key(), pos_integer()}),
+    % we rely on erlang sorting order where `number < atom`
+    min = infinite :: infinite | pos_integer(),
+    max = 0 :: non_neg_integer(),
+    size = 0 :: non_neg_integer(),
+    % capacity cannot be less than 1
+    capacity = 1 :: pos_integer()
+}).
+
+new_topK(K) when K >= 1 ->
+    #topK{capacity = K}.
+
+% when we are at capacity
+% don't bother adding the value since it is less than what we already saw
+update_topK(_Key, Value, #topK{size = S, capacity = S, min = Min} = Top) when Value < Min ->
+    Top#topK{min = Value};
+% when we are at capacity evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, max = Max, seq = Seq} = Top) when
+    Value > Max
+->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{max = Value, seq = lists:keysort(2, [{Key, Value} | Truncated])};
+% when we are at capacity and value is in between min and max evict smallest value
+update_topK(Key, Value, #topK{size = S, capacity = S, seq = Seq} = Top) ->
+    % capacity cannot be less than 1, so we can avoid handling the case when Seq is empty
+    [_ | Truncated] = Seq,
+    Top#topK{seq = lists:keysort(2, [{Key, Value} | Truncated])};
+update_topK(Key, Value, #topK{size = S, min = Min, seq = Seq} = Top) when Value < Min ->
+    Top#topK{size = S + 1, min = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, max = Max, seq = Seq} = Top) when Value > Max ->
+    Top#topK{size = S + 1, max = Value, seq = lists:keysort(2, [{Key, Value} | Seq])};
+update_topK(Key, Value, #topK{size = S, seq = Seq} = Top) ->
+    Top#topK{size = S + 1, seq = lists:keysort(2, [{Key, Value} | Seq])}.

Review Comment:
   I think that's not worth it for this case. It's not in the main data path, it's an API that will be called very rarely by an operator when investing an issue so it's worth keeping the code simple and compact more than adding extra complexity for possible performance gains. If this is a large bottleneck in the future we can always add more optimizations later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org