[cassandra] 01/03: Add documentation for partition denylisting

[sm...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

smiklosovic pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 0b47c57184cebeac54f5858571b880bcb90917c6
Author: Sharan Foga <sh...@apache.org>
AuthorDate: Sun Nov 6 19:14:03 2022 +0100

    Add documentation for partition denylisting
    
    patch by Sharan Foga; reviewed by Stefan Miklosovic and Josh McKenzie for CASSANDRA-17547
---
 .../pages/operating/denylisting_partitions.adoc    | 143 +++++++++++++++++++++
 doc/modules/cassandra/pages/operating/index.adoc   |   1 +
 2 files changed, 144 insertions(+)

diff --git a/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc b/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc
new file mode 100644
index 0000000000..a81dc8009c
--- /dev/null
+++ b/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc
@@ -0,0 +1,143 @@
+= Denylisting Partitions
+
+Due to access patterns and data modeling, sometimes there are specific partitions
+that are "hot" and can cause instability in a Cassandra cluster. This often occurs
+when your data model includes many update or insert operations on a single partition,
+causing the partition to grow very large over time and in turn making it very expensive
+to read and maintain.
+
+Cassandra supports "denylisting" these problematic partitions so that when clients
+issue point reads (`SELECT` statements with the partition key specified) or range
+reads (`SELECT *`, etc that pull a range of data) that intersect with a blocked
+partition key, the query will be immediately rejected with an `InvalidQueryException`.
+
+== How to denylist a partition key
+
+The ``system_distributed.denylisted_partitions`` table can be used to denylist partitions.
+There are a couple of ways to interact with and mutate this data. First: directly
+via CQL by inserting a record with the following details:
+
+- Keyspace name (ks_name)
+- Table name (table_name)
+- Partition Key (partition_key)
+
+The partition key format needs to be in the same form required by ``nodetool getendpoints``.
+
+Following are several examples for denylisting partition keys in keyspace `ks` and
+table `table1` for different data types on the primary key `Id`:
+
+- Id is a simple type - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','1');`
+- Id is a blob        - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','12345f');`
+- Id has a colon      - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','1\:2');`
+
+In the case of composite column partition keys (Key1, Key2):
+
+- `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks', 'table1', 'k11:k21')`
+
+
+=== Special considerations
+
+The denylist has the property in that you want to keep your cache (see below) and
+CQL data on a replica set as close together as possible, so you don't have different
+nodes in your cluster denying or allowing different keys. To best achieve this,
+the workflow for a denylist change (addition or deletion) should always be as follows:
+
+JMX PATH (preferred for single changes):
+
+1. Call the JMX hook for ``denylistKey()`` with the desired key
+2. Double-check the cache reloaded with ``isKeyDenylisted()``
+3. Check for warnings about unrecognized keyspace/table combinations, limits, or
+consistency level. If you get a message about nodes being down and not hitting CL
+for denylist, recover the downed nodes and then trigger a re-load of the cache on each
+node with ``loadPartitionDenylist()``
+
+CQL PATH (preferred for bulk changes):
+
+1. Mutate the denylisted partition lists via CQL
+2. Trigger a re-load of the denylist cache on each node via JMX ``loadPartitionDenylist()`` (see below)
+3. Check for warnings about lack of availability for a denylist refresh. In the event nodes are down, recover them, then go to 2.
+
+Due to conditions on known unavailable range slices leading to alert storming on
+startup, the denylist cache will not load on node start unless it can achieve the
+configured consistency level in `cassandra.yaml` - `denylist_consistency_level`.
+The JMX call to `loadPartitionDenylist` will, however, load the cache regardless
+of the number of nodes available. This leaves the control for denylisting or not
+denylisting during degraded cluster states in the hands of the operator.
+
+== Denylisted Partitions Cache
+
+Cassandra internally maintains an on-heap cache of denylisted partitions loaded
+from ``system_distributed.denylisted_partitions``. The values for a table will be
+automatically repopulated every ``denylist_refresh`` as specified in the
+`conf/cassandra.yaml` file, defaulting to `600s`, or 10 minutes. Invalid records
+(unknown keyspaces, tables, or keys) will be ignored and not cached on load.
+
+The cache can be refreshed in the following ways:
+
+- During Cassandra node startup
+- Via the automatic on-heap cache refresh mechanisms. Note: this will occur asynchronously
+on query after the ``denylist_refresh`` time is hit.
+- Via the JMX command: ``loadPartitionDenylist`` in ``the org.apache.cassandra.service.
+StorageProxyMBean`` invocation point.
+
+The Cache size is bounded by the following two config properties
+
+- denylist_max_keys_per_table
+- denylist_max_keys_total
+
+On cache load, if a table exceeds the value allowed in `denylist_max_keys_per_table` (defaults to 1000),
+a warning will be printed to the logs and the remainder of the keys will not be cached.
+Similarly, if the total allowed size is exceeded, subsequent ks_name + table_name
+combinations (in clustering / lexicographical order) will be skipped as well, and a
+warning logged to the server logs.
+
+[NOTE]
+====
+Given the required workflow of 1) Mutate, 2) Reload cache, the auto-reload
+property seems superfluous. It exists to ensure that, should an operator make a
+mistake and denylist (or undenylist) a key but forget to reload the cache, that
+intent will be captured on the next cache reload.
+====
+
+== JMX Interface
+
+[cols="1,1"]
+|===
+| Command | Effect
+
+| loadPartitionDenylist()
+| Reloads cached denylist from CQL table
+
+| getPartitionDenylistLoadAttempts()
+| Gets the count of cache reload attempts
+
+| getPartitionDenylistLoadSuccesses()
+| Gets the count of cache reload successes
+
+| setEnablePartitionDenylist(boolean enabled)
+| Enables or disables the partition denylisting functionality
+
+| setEnableDenylistWrites(boolean enabled)
+| Enables or disables write denylisting functionality
+
+| setEnableDenylistReads(boolean enabled)
+| Enables or disables read denylisting functionality
+
+| setEnableDenylistRangeReads(boolean enabled)
+| Enables or disables range read denylisting functionality
+
+| denylistKey(String keyspace, String table, String partitionKeyAsString)
+| Adds a specific keyspace, table, and partition key combo to the denylist
+
+| removeDenylistKey(String keyspace, String cf, String partitionKeyAsString)
+| Removes a specific keyspace, table, and partition key combo from the denylist
+
+| setDenylistMaxKeysPerTable(int value)
+| Limits count of allowed keys per table in the denylist
+
+| setDenylistMaxKeysTotal(int value)
+| Limits the total count of allowable denylisted keys in the system
+
+| isKeyDenylisted(String keyspace, String table, String partitionKeyAsString)
+| Indicates whether the keyspace.table has the input partition key denied
+|===
diff --git a/doc/modules/cassandra/pages/operating/index.adoc b/doc/modules/cassandra/pages/operating/index.adoc
index 367430a2ef..41ee1b60ca 100644
--- a/doc/modules/cassandra/pages/operating/index.adoc
+++ b/doc/modules/cassandra/pages/operating/index.adoc
@@ -13,3 +13,4 @@
 * xref:operating/bulk_loading.adoc[Bulk loading]
 * xref:operating/cdc.adoc[CDC]
 * xref:operating/bloom_filters.adoc[Bloom filters]
+* xref:operating/denylisting_partitions.adoc[Denylisting partitions]


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org