You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/08/07 18:41:04 UTC
[GitHub] [cassandra] dcapwell commented on a change in pull request #705: Cassandra-15971 Improve FQL doc page
dcapwell commented on a change in pull request #705:
URL: https://github.com/apache/cassandra/pull/705#discussion_r467190553
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -14,174 +14,169 @@
.. See the License for the specific language governing permissions and
.. limitations under the License.
-Full Query Logging
-------------------
+Full Query Logging (FQL)
+========================
-Apache Cassandra 4.0 adds a new feature to support a means of logging all queries as they were invoked (`CASSANDRA-13983
-<https://issues.apache.org/jira/browse/CASSANDRA-13983>`_). For correctness testing it's useful to be able to capture production traffic so that it can be replayed against both the old and new versions of Cassandra while comparing the results.
+Apache Cassandra 4.0 adds a new highly performant feature that supports live query logging (`CASSANDRA-13983 <https://issues.apache.org/jira/browse/CASSANDRA-13983>`_).
+FQL is safe for production use, with configurable limits to heap memory and disk space to prevent out-of-memory errors.
+This feature is useful for live traffic capture, as well as traffic replay. The tool provided can be used for both debugging query traffic and migration.
+New ``nodetool`` options are also added to enable, disable or reset FQL, as well as a new tool to read and replay the binary logs.
+The full query logging (FQL) capability uses `Chronicle-Queue <http://github.com/OpenHFT/Chronicle-Queue>`_ to rotate a log of queries.
+Full query logs will be referred to as *logs* for the remainder of the page.
-Cassandra 4.0 includes an implementation of a full query logging (FQL) that uses chronicle-queue to implement a rotating log of queries. Some of the features of FQL are:
+Some of the features of FQL are:
-- Single thread asynchronously writes log entries to disk to reduce impact on query latency
-- Heap memory usage bounded by a weighted queue with configurable maximum weight sitting in front of logging thread
-- If the weighted queue is full producers can be blocked or samples can be dropped
-- Disk utilization is bounded by deleting old log segments once a configurable size is reached
-- The on disk serialization uses a flexible schema binary format (chronicle-wire) making it easy to skip unrecognized fields, add new ones, and omit old ones.
-- Can be enabled and configured via JMX, disabled, and reset (delete on disk data), logging path is configurable via both JMX and YAML
-- Introduce new ``fqltool`` in ``/bin`` that currently implements ``Dump`` which can dump in a readable format full query logs as well as follow active full query logs. FQL ``Replay`` and ``Compare`` are also available.
+- The impact on query latency is reduced by asynchronous single-thread log entry writes to disk.
+- Heap memory usage is bounded by a weighted queue, with configurable maximum weight sitting in front of logging thread.
+- If the weighted queue is full, producers can be blocked or samples can be dropped.
+- Disk utilization is bounded by a configurable size, deleting old log segments once the limit is reached.
+- A flexible schema binary format, `Chronicle-Wire <http://github.com/OpenHFT/Chronicle-Wire>`_, for on-disk serialization that can skip unrecognized fields, add new ones, and omit old ones.
+- Can be enabled, disabled, or reset (to delete on-disk data) using the JMX tool, ``nodetool``.
+- Can configure the settings in either the `cassandra.yaml` file or by using ``nodetool``.
+- Introduces new ``fqltool`` that currently can ``Dump`` the binary logs to a readable format. Other options are ``Replay`` and ``Compare``.
-Cassandra 4.0 has a binary full query log based on Chronicle Queue that can be controlled using ``nodetool enablefullquerylog``, ``disablefullquerylog``, and ``resetfullquerylog``. The log contains all queries invoked, approximate time they were invoked, any parameters necessary to bind wildcard values, and all query options. A readable version of the log can be dumped or tailed using the new ``bin/fqltool`` utility. The full query log is designed to be safe to use in production and limits utilization of heap memory and disk space with limits you can specify when enabling the log.
+FQL logs all Cassandra Query Language (CQL) requests, both events that modify the data and those that query.
Review comment:
> FQL logs all Cassandra Query Language (CQL)
should be
> FQL logs all successful Cassandra Query Language (CQL)
Checking the code, bad queries (ones which don't parse or cause an exception) will not be logged.
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -14,174 +14,169 @@
.. See the License for the specific language governing permissions and
.. limitations under the License.
-Full Query Logging
-------------------
+Full Query Logging (FQL)
+========================
-Apache Cassandra 4.0 adds a new feature to support a means of logging all queries as they were invoked (`CASSANDRA-13983
-<https://issues.apache.org/jira/browse/CASSANDRA-13983>`_). For correctness testing it's useful to be able to capture production traffic so that it can be replayed against both the old and new versions of Cassandra while comparing the results.
+Apache Cassandra 4.0 adds a new highly performant feature that supports live query logging (`CASSANDRA-13983 <https://issues.apache.org/jira/browse/CASSANDRA-13983>`_).
+FQL is safe for production use, with configurable limits to heap memory and disk space to prevent out-of-memory errors.
+This feature is useful for live traffic capture, as well as traffic replay. The tool provided can be used for both debugging query traffic and migration.
+New ``nodetool`` options are also added to enable, disable or reset FQL, as well as a new tool to read and replay the binary logs.
+The full query logging (FQL) capability uses `Chronicle-Queue <http://github.com/OpenHFT/Chronicle-Queue>`_ to rotate a log of queries.
+Full query logs will be referred to as *logs* for the remainder of the page.
-Cassandra 4.0 includes an implementation of a full query logging (FQL) that uses chronicle-queue to implement a rotating log of queries. Some of the features of FQL are:
+Some of the features of FQL are:
-- Single thread asynchronously writes log entries to disk to reduce impact on query latency
-- Heap memory usage bounded by a weighted queue with configurable maximum weight sitting in front of logging thread
-- If the weighted queue is full producers can be blocked or samples can be dropped
-- Disk utilization is bounded by deleting old log segments once a configurable size is reached
-- The on disk serialization uses a flexible schema binary format (chronicle-wire) making it easy to skip unrecognized fields, add new ones, and omit old ones.
-- Can be enabled and configured via JMX, disabled, and reset (delete on disk data), logging path is configurable via both JMX and YAML
-- Introduce new ``fqltool`` in ``/bin`` that currently implements ``Dump`` which can dump in a readable format full query logs as well as follow active full query logs. FQL ``Replay`` and ``Compare`` are also available.
+- The impact on query latency is reduced by asynchronous single-thread log entry writes to disk.
+- Heap memory usage is bounded by a weighted queue, with configurable maximum weight sitting in front of logging thread.
+- If the weighted queue is full, producers can be blocked or samples can be dropped.
+- Disk utilization is bounded by a configurable size, deleting old log segments once the limit is reached.
+- A flexible schema binary format, `Chronicle-Wire <http://github.com/OpenHFT/Chronicle-Wire>`_, for on-disk serialization that can skip unrecognized fields, add new ones, and omit old ones.
+- Can be enabled, disabled, or reset (to delete on-disk data) using the JMX tool, ``nodetool``.
+- Can configure the settings in either the `cassandra.yaml` file or by using ``nodetool``.
+- Introduces new ``fqltool`` that currently can ``Dump`` the binary logs to a readable format. Other options are ``Replay`` and ``Compare``.
-Cassandra 4.0 has a binary full query log based on Chronicle Queue that can be controlled using ``nodetool enablefullquerylog``, ``disablefullquerylog``, and ``resetfullquerylog``. The log contains all queries invoked, approximate time they were invoked, any parameters necessary to bind wildcard values, and all query options. A readable version of the log can be dumped or tailed using the new ``bin/fqltool`` utility. The full query log is designed to be safe to use in production and limits utilization of heap memory and disk space with limits you can specify when enabling the log.
+FQL logs all Cassandra Query Language (CQL) requests, both events that modify the data and those that query.
+While audit logs also include CQL requests, FQL logs only the CQL request. This difference means that FQL can be used to replay or compare logs, which audit logging cannot. FQL is useful for debugging, performance benchmarking, testing and auditing CQL queries, while audit logs are useful for compliance.
-Objective
-^^^^^^^^^^
-Full Query Logging logs all requests to the CQL interface. The full query logs could be used for debugging, performance benchmarking, testing and auditing CQL queries. The audit logs also include CQL requests but full query logging is dedicated to CQL requests only with features such as FQL Replay and FQL Compare that are not available in audit logging.
-
-Full Query Logger
-^^^^^^^^^^^^^^^^^^
-The Full Query Logger is a logger that logs entire query contents after the query finishes. FQL only logs the queries that successfully complete. The other queries (e.g. timed out, failed) are not to be logged. Queries are logged in one of two modes: single query or batch of queries. The log for an invocation of a batch of queries includes the following attributes:
-
-::
-
- type - The type of the batch
- queries - CQL text of the queries
- values - Values to bind to as parameters for the queries
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+In performance testing, FQL appears to have little or no overhead in ``WRITE`` only workloads, and a minor overhead in ``MIXED`` workload.
-The log for single CQL query includes the following attributes:
+Query information logged
+------------------------
-::
-
- query - CQL query text
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- queryTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+The query log contains:
-Full query logging is backed up by ``BinLog``. BinLog is a quick and dirty binary log. Its goal is good enough performance, predictable footprint, simplicity in terms of implementation and configuration and most importantly minimal impact on producers of log records. Performance safety is accomplished by feeding items to the binary log using a weighted queue and dropping records if the binary log falls sufficiently far behind. Simplicity and good enough performance is achieved by using a single log writing thread as well as Chronicle Queue to handle writing the log, making it available for readers, as well as log rolling.
+- all queries invoked
+- approximate time they were invoked
+- any parameters necessary to bind wildcard values
+- all query options
-Weighted queue is a wrapper around any blocking queue that turns it into a blocking weighted queue. The queue will weigh each element being added and removed. Adding to the queue is blocked if adding would violate the weight bound. If an element weighs in at larger than the capacity of the queue then exactly one such element will be allowed into the queue at a time. If the weight of an object changes after it is added it could create issues. Checking weight should be cheap so memorize expensive to compute weights. If weight throws that can also result in leaked permits so it's always a good idea to memorize weight so it doesn't throw. In the interests of not writing unit tests for methods no one uses there is a lot of ``UnsupportedOperationException``. If you need them then add them and add proper unit tests to ``WeightedQueueTest``. "Good" tests. 100% coverage including exception paths and resource leaks.
+The logger writes single or batched CQL queries after they finish, so only successfully completed queries are logged. Failed or timed-out queries are not logged. Different data is logged, depending on the type of query.
+A single CQL query log entry contains:
-The FQL tracks information about store files:
+- query - CQL query text
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- queryTimeMillis - Approximate time in milliseconds since the epoch since the query was invoked
-- Store files as they are added and their storage impact. Delete them if over storage limit.
-- The files in the chronicle queue that have already rolled
-- The number of bytes in store files that have already rolled
+A batch CQL query log entry contains:
-FQL logger sequence is as follows:
+- queries - CQL text of the queries
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+- type - The type of the batch
+- values - Values to bind to as parameters for the queries
-1. Start the consumer thread that writes log records. Can only be done once.
-2. Offer a record to the log. If the in memory queue is full the record will be dropped and offer will return false.
-3. Put a record into the log. If the in memory queue is full the putting thread will be blocked until there is space or it is interrupted.
-4. Clean up the buffers on thread exit, finalization will check again once this is no longer reachable ensuring there are no stragglers in the queue.
-5. Stop the consumer thread that writes log records. Can be called multiple times.
+Because FQL is backed by `Binlog`, the performance and footprint are predictable, with minimal impact on log record producers.
+Performance safety prevents the producers from overloading the log, using a weighted queue to drop records if the logging falls behind.
+Single-thread asynchronous writing produces the logs. Chronicle-Queue provides an easy method of rolling the logs.
-Next, we shall demonstrate full query logging with an example.
+Logging information logged
+--------------------------
+FQL also tracks information about the stored log files:
-Configuring Full Query Logging
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Stored log files that are added and their storage impact. Deletes them if over storage limit.
+- The log files in Chronicle-Queue that have already rolled
+- The number of bytes in the log files that have already rolled
-Full Query Logger default options are configured on a per node basis in ``cassandra.yaml`` with following configuration property.
+Logging sequence
+----------------
-::
+The logger follows a well-defined sequence of events:
- full_query_logging_options:
+1. The consumer thread that writes log records is started. This action can occur only once.
+2. The consumer thread offers a record to the log. If the in-memory queue is full, the record will be dropped and offer returns a `false` value.
+3. If accepted, the record is entered into the log. If the in-memory queue is full, the putting thread will be blocked until there is space or it is interrupted.
+4. The buffers are cleaned up at thread exit. Finalization will check again, to ensure there are no stragglers in the queue.
+5. The consumer thread is stopped. It can be called multiple times.
-As an example setup create a three node Cassandra 4.0 cluster. The ``nodetool status`` command lists the nodes in the cluster.
-
-::
+Using FQL
+---------
- [ec2-user@ip-10-0-2-238 ~]$ nodetool status
- Datacenter: us-east-1
- =====================
- Status=Up/Down
- |/ State=Normal/Leaving/Joining/Moving
- -- AddressLoad Tokens Owns (effective) Host ID Rack
- UN 10.0.1.115 442.42 KiB 25632.6% b64cb32a-b32a-46b4-9eeb-e123fa8fc287 us-east-1b
- UN 10.0.3.206 559.52 KiB 25631.9% 74863177-684b-45f4-99f7-d1006625dc9e us-east-1d
- UN 10.0.2.238 587.87 KiB 25635.5% 4dcdadd2-41f9-4f34-9892-1f20868b27c7 us-east-1c
+To use FQL, two actions must be completed. FQL must be configured using either the `cassandra.yaml` file or ``nodetool``, and logging must be enabled using ``nodetool enablefullquerylog``.
+Both actions are completed on a per-node basis.
+With either method, at a minimum, the path to the log directory must be specified.
+Full query logs are generated on each enabled node, so logs on each node will have that node's queries.
+Configuring FQL in cassandra.yaml
+---------------------------------
-In subsequent sub-sections we shall discuss enabling and configuring full query logging.
+The `cassandra.yaml` file can be used to configure FQL before enabling the feature with ``nodetool``.
-Setting the FQL Directory
-*************************
+The file includes the following options that can be uncommented for use:
-A dedicated directory path must be provided to write full query log data to when the full query log is enabled. The directory for FQL must exist, and have permissions set. The full query log will recursively delete the contents of this path at times. It is recommended not to place links in this directory to other sections of the filesystem. The ``full_query_log_dir`` property in ``cassandra.yaml`` is pre-configured.
+::
-::
-
- full_query_log_dir: /tmp/cassandrafullquerylog
-
-The ``log_dir`` option may be used to configure the FQL directory if the ``full_query_log_dir`` is not set.
-
-::
-
- full_query_logging_options:
+ # default options for full query logging - these can be overridden from command line
+ # when executing nodetool enablefullquerylog
+ #full_query_logging_options:
# log_dir:
+ # roll_cycle: HOURLY
+ # block: true
+ # max_queue_weight: 268435456 # 256 MiB
+ # max_log_size: 17179869184 # 16 GiB
+ ## archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled:
+ # archive_command:
+ # max_archive_retries: 10
-Create the FQL directory if it does not exist and set its permissions.
+log_dir
+^^^^^^^
-::
+To write logs, an existing directory must be set in ``log_dir``.
- sudo mkdir -p /tmp/cassandrafullquerylog
- sudo chmod -R 777 /tmp/cassandrafullquerylog
+The directory must have appropriate permissions set to allow reading, writing, and executing.
+Logging will recursively delete the directory contents as needed.
+Do not place links in this directory to other sections of the filesystem.
+For example, ``log_dir: /tmp/cassandrafullquerylog``.
-Setting the Roll Cycle
-**********************
+roll_cycle
+^^^^^^^^^^
-The ``roll_cycle`` option sets how often to roll FQL log segments so they can potentially be reclaimed. Supported values are ``MINUTELY``, ``HOURLY`` and ``DAILY``. Default setting is ``HOURLY``.
+The ``roll_cycle`` defines the frequency with which the log segments are rolled.
+Supported values are ``HOURLY`` (default), ``MINUTELY``, and ``DAILY``.
+For example: ``roll_cycle: DAILY``
-::
+block
+^^^^^
- roll_cycle: HOURLY
+The ``block`` option specifies whether FQL should block writing or drop log records if FQL falls behind. Supported boolean values are ``true`` (default) or ``false``.
+For example: ``block: false`` to drop records
-Setting Other Options
-*********************
+max_queue_weight
+^^^^^^^^^^^^^^^^
-The ``block`` option specifies whether the FQL should block if the FQL falls behind or should drop log records. Default value of ``block`` is ``true``. The ``max_queue_weight`` option sets the maximum weight of in memory queue for records waiting to be written to the file before blocking or dropping. The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The ``archive_command`` option sets the archive command to execute on rolled log files. The ``max_archive_retries`` option sets the max number of retries of failed archive commands.
+The ``max_queue_weight`` option sets the maximum weight of in-memory queue for records waiting to be written to the file before blocking or dropping. The option must be set to a positive value. The default value is 268435456, or 256 MiB.
+For example, to change the default: ``max_queue_weight: 134217728 # 128 MiB``
-::
+max_log_size
+^^^^^^^^^^^^
- # block: true
- # max_queue_weight: 268435456 # 256 MiB
- # max_log_size: 17179869184 # 16 GiB
- ## archive command is "/path/to/script.sh %path" where %path is replaced with the file
- being rolled:
- # archive_command:
- # max_archive_retries: 10
+The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The option must be set to a positive value. The default is 17179869184, or 16 GiB.
+For example, to change the default: ``max_log_size: 34359738368 # 32 GiB``
-The ``max_queue_weight`` must be > 0. Similarly ``max_log_size`` must be > 0. An example full query logging options is as follows.
+archive_command
+^^^^^^^^^^^^^^^
-::
+The ``archive_command`` option sets the user-defined archive script to execute on rolled log files.
Review comment:
should state that when not defined, files are deleted
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -14,174 +14,169 @@
.. See the License for the specific language governing permissions and
.. limitations under the License.
-Full Query Logging
-------------------
+Full Query Logging (FQL)
+========================
-Apache Cassandra 4.0 adds a new feature to support a means of logging all queries as they were invoked (`CASSANDRA-13983
-<https://issues.apache.org/jira/browse/CASSANDRA-13983>`_). For correctness testing it's useful to be able to capture production traffic so that it can be replayed against both the old and new versions of Cassandra while comparing the results.
+Apache Cassandra 4.0 adds a new highly performant feature that supports live query logging (`CASSANDRA-13983 <https://issues.apache.org/jira/browse/CASSANDRA-13983>`_).
+FQL is safe for production use, with configurable limits to heap memory and disk space to prevent out-of-memory errors.
+This feature is useful for live traffic capture, as well as traffic replay. The tool provided can be used for both debugging query traffic and migration.
+New ``nodetool`` options are also added to enable, disable or reset FQL, as well as a new tool to read and replay the binary logs.
+The full query logging (FQL) capability uses `Chronicle-Queue <http://github.com/OpenHFT/Chronicle-Queue>`_ to rotate a log of queries.
+Full query logs will be referred to as *logs* for the remainder of the page.
-Cassandra 4.0 includes an implementation of a full query logging (FQL) that uses chronicle-queue to implement a rotating log of queries. Some of the features of FQL are:
+Some of the features of FQL are:
-- Single thread asynchronously writes log entries to disk to reduce impact on query latency
-- Heap memory usage bounded by a weighted queue with configurable maximum weight sitting in front of logging thread
-- If the weighted queue is full producers can be blocked or samples can be dropped
-- Disk utilization is bounded by deleting old log segments once a configurable size is reached
-- The on disk serialization uses a flexible schema binary format (chronicle-wire) making it easy to skip unrecognized fields, add new ones, and omit old ones.
-- Can be enabled and configured via JMX, disabled, and reset (delete on disk data), logging path is configurable via both JMX and YAML
-- Introduce new ``fqltool`` in ``/bin`` that currently implements ``Dump`` which can dump in a readable format full query logs as well as follow active full query logs. FQL ``Replay`` and ``Compare`` are also available.
+- The impact on query latency is reduced by asynchronous single-thread log entry writes to disk.
+- Heap memory usage is bounded by a weighted queue, with configurable maximum weight sitting in front of logging thread.
+- If the weighted queue is full, producers can be blocked or samples can be dropped.
+- Disk utilization is bounded by a configurable size, deleting old log segments once the limit is reached.
+- A flexible schema binary format, `Chronicle-Wire <http://github.com/OpenHFT/Chronicle-Wire>`_, for on-disk serialization that can skip unrecognized fields, add new ones, and omit old ones.
+- Can be enabled, disabled, or reset (to delete on-disk data) using the JMX tool, ``nodetool``.
+- Can configure the settings in either the `cassandra.yaml` file or by using ``nodetool``.
+- Introduces new ``fqltool`` that currently can ``Dump`` the binary logs to a readable format. Other options are ``Replay`` and ``Compare``.
-Cassandra 4.0 has a binary full query log based on Chronicle Queue that can be controlled using ``nodetool enablefullquerylog``, ``disablefullquerylog``, and ``resetfullquerylog``. The log contains all queries invoked, approximate time they were invoked, any parameters necessary to bind wildcard values, and all query options. A readable version of the log can be dumped or tailed using the new ``bin/fqltool`` utility. The full query log is designed to be safe to use in production and limits utilization of heap memory and disk space with limits you can specify when enabling the log.
+FQL logs all Cassandra Query Language (CQL) requests, both events that modify the data and those that query.
+While audit logs also include CQL requests, FQL logs only the CQL request. This difference means that FQL can be used to replay or compare logs, which audit logging cannot. FQL is useful for debugging, performance benchmarking, testing and auditing CQL queries, while audit logs are useful for compliance.
-Objective
-^^^^^^^^^^
-Full Query Logging logs all requests to the CQL interface. The full query logs could be used for debugging, performance benchmarking, testing and auditing CQL queries. The audit logs also include CQL requests but full query logging is dedicated to CQL requests only with features such as FQL Replay and FQL Compare that are not available in audit logging.
-
-Full Query Logger
-^^^^^^^^^^^^^^^^^^
-The Full Query Logger is a logger that logs entire query contents after the query finishes. FQL only logs the queries that successfully complete. The other queries (e.g. timed out, failed) are not to be logged. Queries are logged in one of two modes: single query or batch of queries. The log for an invocation of a batch of queries includes the following attributes:
-
-::
-
- type - The type of the batch
- queries - CQL text of the queries
- values - Values to bind to as parameters for the queries
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+In performance testing, FQL appears to have little or no overhead in ``WRITE`` only workloads, and a minor overhead in ``MIXED`` workload.
-The log for single CQL query includes the following attributes:
+Query information logged
+------------------------
-::
-
- query - CQL query text
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- queryTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+The query log contains:
-Full query logging is backed up by ``BinLog``. BinLog is a quick and dirty binary log. Its goal is good enough performance, predictable footprint, simplicity in terms of implementation and configuration and most importantly minimal impact on producers of log records. Performance safety is accomplished by feeding items to the binary log using a weighted queue and dropping records if the binary log falls sufficiently far behind. Simplicity and good enough performance is achieved by using a single log writing thread as well as Chronicle Queue to handle writing the log, making it available for readers, as well as log rolling.
+- all queries invoked
+- approximate time they were invoked
+- any parameters necessary to bind wildcard values
+- all query options
-Weighted queue is a wrapper around any blocking queue that turns it into a blocking weighted queue. The queue will weigh each element being added and removed. Adding to the queue is blocked if adding would violate the weight bound. If an element weighs in at larger than the capacity of the queue then exactly one such element will be allowed into the queue at a time. If the weight of an object changes after it is added it could create issues. Checking weight should be cheap so memorize expensive to compute weights. If weight throws that can also result in leaked permits so it's always a good idea to memorize weight so it doesn't throw. In the interests of not writing unit tests for methods no one uses there is a lot of ``UnsupportedOperationException``. If you need them then add them and add proper unit tests to ``WeightedQueueTest``. "Good" tests. 100% coverage including exception paths and resource leaks.
+The logger writes single or batched CQL queries after they finish, so only successfully completed queries are logged. Failed or timed-out queries are not logged. Different data is logged, depending on the type of query.
+A single CQL query log entry contains:
-The FQL tracks information about store files:
+- query - CQL query text
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- queryTimeMillis - Approximate time in milliseconds since the epoch since the query was invoked
-- Store files as they are added and their storage impact. Delete them if over storage limit.
-- The files in the chronicle queue that have already rolled
-- The number of bytes in store files that have already rolled
+A batch CQL query log entry contains:
-FQL logger sequence is as follows:
+- queries - CQL text of the queries
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+- type - The type of the batch
+- values - Values to bind to as parameters for the queries
-1. Start the consumer thread that writes log records. Can only be done once.
-2. Offer a record to the log. If the in memory queue is full the record will be dropped and offer will return false.
-3. Put a record into the log. If the in memory queue is full the putting thread will be blocked until there is space or it is interrupted.
-4. Clean up the buffers on thread exit, finalization will check again once this is no longer reachable ensuring there are no stragglers in the queue.
-5. Stop the consumer thread that writes log records. Can be called multiple times.
+Because FQL is backed by `Binlog`, the performance and footprint are predictable, with minimal impact on log record producers.
+Performance safety prevents the producers from overloading the log, using a weighted queue to drop records if the logging falls behind.
+Single-thread asynchronous writing produces the logs. Chronicle-Queue provides an easy method of rolling the logs.
-Next, we shall demonstrate full query logging with an example.
+Logging information logged
+--------------------------
+FQL also tracks information about the stored log files:
-Configuring Full Query Logging
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Stored log files that are added and their storage impact. Deletes them if over storage limit.
+- The log files in Chronicle-Queue that have already rolled
+- The number of bytes in the log files that have already rolled
-Full Query Logger default options are configured on a per node basis in ``cassandra.yaml`` with following configuration property.
+Logging sequence
+----------------
-::
+The logger follows a well-defined sequence of events:
- full_query_logging_options:
+1. The consumer thread that writes log records is started. This action can occur only once.
+2. The consumer thread offers a record to the log. If the in-memory queue is full, the record will be dropped and offer returns a `false` value.
+3. If accepted, the record is entered into the log. If the in-memory queue is full, the putting thread will be blocked until there is space or it is interrupted.
+4. The buffers are cleaned up at thread exit. Finalization will check again, to ensure there are no stragglers in the queue.
+5. The consumer thread is stopped. It can be called multiple times.
-As an example setup create a three node Cassandra 4.0 cluster. The ``nodetool status`` command lists the nodes in the cluster.
-
-::
+Using FQL
+---------
- [ec2-user@ip-10-0-2-238 ~]$ nodetool status
- Datacenter: us-east-1
- =====================
- Status=Up/Down
- |/ State=Normal/Leaving/Joining/Moving
- -- AddressLoad Tokens Owns (effective) Host ID Rack
- UN 10.0.1.115 442.42 KiB 25632.6% b64cb32a-b32a-46b4-9eeb-e123fa8fc287 us-east-1b
- UN 10.0.3.206 559.52 KiB 25631.9% 74863177-684b-45f4-99f7-d1006625dc9e us-east-1d
- UN 10.0.2.238 587.87 KiB 25635.5% 4dcdadd2-41f9-4f34-9892-1f20868b27c7 us-east-1c
+To use FQL, two actions must be completed. FQL must be configured using either the `cassandra.yaml` file or ``nodetool``, and logging must be enabled using ``nodetool enablefullquerylog``.
+Both actions are completed on a per-node basis.
+With either method, at a minimum, the path to the log directory must be specified.
+Full query logs are generated on each enabled node, so logs on each node will have that node's queries.
+Configuring FQL in cassandra.yaml
+---------------------------------
-In subsequent sub-sections we shall discuss enabling and configuring full query logging.
+The `cassandra.yaml` file can be used to configure FQL before enabling the feature with ``nodetool``.
-Setting the FQL Directory
-*************************
+The file includes the following options that can be uncommented for use:
-A dedicated directory path must be provided to write full query log data to when the full query log is enabled. The directory for FQL must exist, and have permissions set. The full query log will recursively delete the contents of this path at times. It is recommended not to place links in this directory to other sections of the filesystem. The ``full_query_log_dir`` property in ``cassandra.yaml`` is pre-configured.
+::
-::
-
- full_query_log_dir: /tmp/cassandrafullquerylog
-
-The ``log_dir`` option may be used to configure the FQL directory if the ``full_query_log_dir`` is not set.
-
-::
-
- full_query_logging_options:
+ # default options for full query logging - these can be overridden from command line
+ # when executing nodetool enablefullquerylog
+ #full_query_logging_options:
# log_dir:
+ # roll_cycle: HOURLY
+ # block: true
+ # max_queue_weight: 268435456 # 256 MiB
+ # max_log_size: 17179869184 # 16 GiB
+ ## archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled:
+ # archive_command:
+ # max_archive_retries: 10
-Create the FQL directory if it does not exist and set its permissions.
+log_dir
+^^^^^^^
-::
+To write logs, an existing directory must be set in ``log_dir``.
- sudo mkdir -p /tmp/cassandrafullquerylog
- sudo chmod -R 777 /tmp/cassandrafullquerylog
+The directory must have appropriate permissions set to allow reading, writing, and executing.
+Logging will recursively delete the directory contents as needed.
+Do not place links in this directory to other sections of the filesystem.
+For example, ``log_dir: /tmp/cassandrafullquerylog``.
-Setting the Roll Cycle
-**********************
+roll_cycle
+^^^^^^^^^^
-The ``roll_cycle`` option sets how often to roll FQL log segments so they can potentially be reclaimed. Supported values are ``MINUTELY``, ``HOURLY`` and ``DAILY``. Default setting is ``HOURLY``.
+The ``roll_cycle`` defines the frequency with which the log segments are rolled.
+Supported values are ``HOURLY`` (default), ``MINUTELY``, and ``DAILY``.
+For example: ``roll_cycle: DAILY``
-::
+block
+^^^^^
- roll_cycle: HOURLY
+The ``block`` option specifies whether FQL should block writing or drop log records if FQL falls behind. Supported boolean values are ``true`` (default) or ``false``.
+For example: ``block: false`` to drop records
-Setting Other Options
-*********************
+max_queue_weight
+^^^^^^^^^^^^^^^^
-The ``block`` option specifies whether the FQL should block if the FQL falls behind or should drop log records. Default value of ``block`` is ``true``. The ``max_queue_weight`` option sets the maximum weight of in memory queue for records waiting to be written to the file before blocking or dropping. The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The ``archive_command`` option sets the archive command to execute on rolled log files. The ``max_archive_retries`` option sets the max number of retries of failed archive commands.
+The ``max_queue_weight`` option sets the maximum weight of in-memory queue for records waiting to be written to the file before blocking or dropping. The option must be set to a positive value. The default value is 268435456, or 256 MiB.
+For example, to change the default: ``max_queue_weight: 134217728 # 128 MiB``
-::
+max_log_size
+^^^^^^^^^^^^
- # block: true
- # max_queue_weight: 268435456 # 256 MiB
- # max_log_size: 17179869184 # 16 GiB
- ## archive command is "/path/to/script.sh %path" where %path is replaced with the file
- being rolled:
- # archive_command:
- # max_archive_retries: 10
+The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The option must be set to a positive value. The default is 17179869184, or 16 GiB.
+For example, to change the default: ``max_log_size: 34359738368 # 32 GiB``
-The ``max_queue_weight`` must be > 0. Similarly ``max_log_size`` must be > 0. An example full query logging options is as follows.
+archive_command
+^^^^^^^^^^^^^^^
-::
+The ``archive_command`` option sets the user-defined archive script to execute on rolled log files.
+For example: ``archive_command: /usr/local/bin/archiveit.sh %path # %path is the file being rolled``
- full_query_log_dir: /tmp/cassandrafullquerylog
+max_archive_retries
+^^^^^^^^^^^^^^^^^^^
- # default options for full query logging - these can be overridden from command line when
- executing
- # nodetool enablefullquerylog
- # nodetool enablefullquerylog
- #full_query_logging_options:
- # log_dir:
- roll_cycle: HOURLY
- # block: true
- # max_queue_weight: 268435456 # 256 MiB
- # max_log_size: 17179869184 # 16 GiB
- ## archive command is "/path/to/script.sh %path" where %path is replaced with the file
- being rolled:
- # archive_command:
- # max_archive_retries: 10
+The ``max_archive_retries`` option sets the max number of retries of failed archive commands. The default is 10.
+For example: ``max_archive_retries: 10``
-The ``full_query_log_dir`` setting is not within the ``full_query_logging_options`` but still is for full query logging.
+FQL can also be configured using ``nodetool` when enabling the feature, and will override any values set in the `cassandra.yaml` file, as discussed in the next section.
Review comment:
nodetool has 2 ` upfront, should be 1
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -239,74 +234,186 @@ The syntax of the nodetool enablefullquerylog command is as follows:
-u <username>, --username <username>
Remote jmx agent username
-Run the following command on each node in the cluster.
+To enable FQL, run the following command on each node in the cluster on which you want to enable logging:
::
nodetool enablefullquerylog --path /tmp/cassandrafullquerylog
-After the full query logging has been enabled run some CQL statements to generate full query logs.
+Disabling or resetting FQL
+-------------
+
+Use the ``nodetool disablefullquerylog`` to disable logging.
+Use ``nodetool resetfullquerylog`` to stop FQL and clear the log files in the configured directory.
+**IMPORTANT:** Using ``nodetool resetfullquerylog`` will delete the log files! Do not use this command unless you need to delete all log files.
+
+fqltool
+-------
+
+The ``fqltool`` command is used to view (dump), replay, or compare logs.
+``fqltool dump`` converts the binary log files into human-readable format; only the log directory must be supplied as a command-line option.
+
+``fqltool replay`` (CASSANDRA-14618) enables replay of logs.
+The command can run from a different machine or cluster for testing, debugging, or performance benchmarking.
+The command, run on the same node on which the logs are generated can recreate a dropped database object.
Review comment:
> The command, run on the same node on which the logs are generated can recreate a dropped database object
I don't know what this means, what are you trying to say?
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -239,74 +234,186 @@ The syntax of the nodetool enablefullquerylog command is as follows:
-u <username>, --username <username>
Remote jmx agent username
-Run the following command on each node in the cluster.
+To enable FQL, run the following command on each node in the cluster on which you want to enable logging:
::
nodetool enablefullquerylog --path /tmp/cassandrafullquerylog
-After the full query logging has been enabled run some CQL statements to generate full query logs.
+Disabling or resetting FQL
+-------------
+
+Use the ``nodetool disablefullquerylog`` to disable logging.
+Use ``nodetool resetfullquerylog`` to stop FQL and clear the log files in the configured directory.
+**IMPORTANT:** Using ``nodetool resetfullquerylog`` will delete the log files! Do not use this command unless you need to delete all log files.
+
+fqltool
+-------
+
+The ``fqltool`` command is used to view (dump), replay, or compare logs.
+``fqltool dump`` converts the binary log files into human-readable format; only the log directory must be supplied as a command-line option.
+
+``fqltool replay`` (CASSANDRA-14618) enables replay of logs.
Review comment:
should this link to `https://issues.apache.org/jira/browse/CASSANDRA-14618`? Previous examples did
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -14,174 +14,169 @@
.. See the License for the specific language governing permissions and
.. limitations under the License.
-Full Query Logging
-------------------
+Full Query Logging (FQL)
+========================
-Apache Cassandra 4.0 adds a new feature to support a means of logging all queries as they were invoked (`CASSANDRA-13983
-<https://issues.apache.org/jira/browse/CASSANDRA-13983>`_). For correctness testing it's useful to be able to capture production traffic so that it can be replayed against both the old and new versions of Cassandra while comparing the results.
+Apache Cassandra 4.0 adds a new highly performant feature that supports live query logging (`CASSANDRA-13983 <https://issues.apache.org/jira/browse/CASSANDRA-13983>`_).
+FQL is safe for production use, with configurable limits to heap memory and disk space to prevent out-of-memory errors.
+This feature is useful for live traffic capture, as well as traffic replay. The tool provided can be used for both debugging query traffic and migration.
+New ``nodetool`` options are also added to enable, disable or reset FQL, as well as a new tool to read and replay the binary logs.
+The full query logging (FQL) capability uses `Chronicle-Queue <http://github.com/OpenHFT/Chronicle-Queue>`_ to rotate a log of queries.
+Full query logs will be referred to as *logs* for the remainder of the page.
-Cassandra 4.0 includes an implementation of a full query logging (FQL) that uses chronicle-queue to implement a rotating log of queries. Some of the features of FQL are:
+Some of the features of FQL are:
-- Single thread asynchronously writes log entries to disk to reduce impact on query latency
-- Heap memory usage bounded by a weighted queue with configurable maximum weight sitting in front of logging thread
-- If the weighted queue is full producers can be blocked or samples can be dropped
-- Disk utilization is bounded by deleting old log segments once a configurable size is reached
-- The on disk serialization uses a flexible schema binary format (chronicle-wire) making it easy to skip unrecognized fields, add new ones, and omit old ones.
-- Can be enabled and configured via JMX, disabled, and reset (delete on disk data), logging path is configurable via both JMX and YAML
-- Introduce new ``fqltool`` in ``/bin`` that currently implements ``Dump`` which can dump in a readable format full query logs as well as follow active full query logs. FQL ``Replay`` and ``Compare`` are also available.
+- The impact on query latency is reduced by asynchronous single-thread log entry writes to disk.
+- Heap memory usage is bounded by a weighted queue, with configurable maximum weight sitting in front of logging thread.
+- If the weighted queue is full, producers can be blocked or samples can be dropped.
+- Disk utilization is bounded by a configurable size, deleting old log segments once the limit is reached.
+- A flexible schema binary format, `Chronicle-Wire <http://github.com/OpenHFT/Chronicle-Wire>`_, for on-disk serialization that can skip unrecognized fields, add new ones, and omit old ones.
+- Can be enabled, disabled, or reset (to delete on-disk data) using the JMX tool, ``nodetool``.
+- Can configure the settings in either the `cassandra.yaml` file or by using ``nodetool``.
+- Introduces new ``fqltool`` that currently can ``Dump`` the binary logs to a readable format. Other options are ``Replay`` and ``Compare``.
-Cassandra 4.0 has a binary full query log based on Chronicle Queue that can be controlled using ``nodetool enablefullquerylog``, ``disablefullquerylog``, and ``resetfullquerylog``. The log contains all queries invoked, approximate time they were invoked, any parameters necessary to bind wildcard values, and all query options. A readable version of the log can be dumped or tailed using the new ``bin/fqltool`` utility. The full query log is designed to be safe to use in production and limits utilization of heap memory and disk space with limits you can specify when enabling the log.
+FQL logs all Cassandra Query Language (CQL) requests, both events that modify the data and those that query.
Review comment:
see you say this later with ` Failed or timed-out queries are not logged.`, still good to be clear
##########
File path: doc/source/new/fqllogging.rst
##########
@@ -14,174 +14,169 @@
.. See the License for the specific language governing permissions and
.. limitations under the License.
-Full Query Logging
-------------------
+Full Query Logging (FQL)
+========================
-Apache Cassandra 4.0 adds a new feature to support a means of logging all queries as they were invoked (`CASSANDRA-13983
-<https://issues.apache.org/jira/browse/CASSANDRA-13983>`_). For correctness testing it's useful to be able to capture production traffic so that it can be replayed against both the old and new versions of Cassandra while comparing the results.
+Apache Cassandra 4.0 adds a new highly performant feature that supports live query logging (`CASSANDRA-13983 <https://issues.apache.org/jira/browse/CASSANDRA-13983>`_).
+FQL is safe for production use, with configurable limits to heap memory and disk space to prevent out-of-memory errors.
+This feature is useful for live traffic capture, as well as traffic replay. The tool provided can be used for both debugging query traffic and migration.
+New ``nodetool`` options are also added to enable, disable or reset FQL, as well as a new tool to read and replay the binary logs.
+The full query logging (FQL) capability uses `Chronicle-Queue <http://github.com/OpenHFT/Chronicle-Queue>`_ to rotate a log of queries.
+Full query logs will be referred to as *logs* for the remainder of the page.
-Cassandra 4.0 includes an implementation of a full query logging (FQL) that uses chronicle-queue to implement a rotating log of queries. Some of the features of FQL are:
+Some of the features of FQL are:
-- Single thread asynchronously writes log entries to disk to reduce impact on query latency
-- Heap memory usage bounded by a weighted queue with configurable maximum weight sitting in front of logging thread
-- If the weighted queue is full producers can be blocked or samples can be dropped
-- Disk utilization is bounded by deleting old log segments once a configurable size is reached
-- The on disk serialization uses a flexible schema binary format (chronicle-wire) making it easy to skip unrecognized fields, add new ones, and omit old ones.
-- Can be enabled and configured via JMX, disabled, and reset (delete on disk data), logging path is configurable via both JMX and YAML
-- Introduce new ``fqltool`` in ``/bin`` that currently implements ``Dump`` which can dump in a readable format full query logs as well as follow active full query logs. FQL ``Replay`` and ``Compare`` are also available.
+- The impact on query latency is reduced by asynchronous single-thread log entry writes to disk.
+- Heap memory usage is bounded by a weighted queue, with configurable maximum weight sitting in front of logging thread.
+- If the weighted queue is full, producers can be blocked or samples can be dropped.
+- Disk utilization is bounded by a configurable size, deleting old log segments once the limit is reached.
+- A flexible schema binary format, `Chronicle-Wire <http://github.com/OpenHFT/Chronicle-Wire>`_, for on-disk serialization that can skip unrecognized fields, add new ones, and omit old ones.
+- Can be enabled, disabled, or reset (to delete on-disk data) using the JMX tool, ``nodetool``.
+- Can configure the settings in either the `cassandra.yaml` file or by using ``nodetool``.
+- Introduces new ``fqltool`` that currently can ``Dump`` the binary logs to a readable format. Other options are ``Replay`` and ``Compare``.
-Cassandra 4.0 has a binary full query log based on Chronicle Queue that can be controlled using ``nodetool enablefullquerylog``, ``disablefullquerylog``, and ``resetfullquerylog``. The log contains all queries invoked, approximate time they were invoked, any parameters necessary to bind wildcard values, and all query options. A readable version of the log can be dumped or tailed using the new ``bin/fqltool`` utility. The full query log is designed to be safe to use in production and limits utilization of heap memory and disk space with limits you can specify when enabling the log.
+FQL logs all Cassandra Query Language (CQL) requests, both events that modify the data and those that query.
+While audit logs also include CQL requests, FQL logs only the CQL request. This difference means that FQL can be used to replay or compare logs, which audit logging cannot. FQL is useful for debugging, performance benchmarking, testing and auditing CQL queries, while audit logs are useful for compliance.
-Objective
-^^^^^^^^^^
-Full Query Logging logs all requests to the CQL interface. The full query logs could be used for debugging, performance benchmarking, testing and auditing CQL queries. The audit logs also include CQL requests but full query logging is dedicated to CQL requests only with features such as FQL Replay and FQL Compare that are not available in audit logging.
-
-Full Query Logger
-^^^^^^^^^^^^^^^^^^
-The Full Query Logger is a logger that logs entire query contents after the query finishes. FQL only logs the queries that successfully complete. The other queries (e.g. timed out, failed) are not to be logged. Queries are logged in one of two modes: single query or batch of queries. The log for an invocation of a batch of queries includes the following attributes:
-
-::
-
- type - The type of the batch
- queries - CQL text of the queries
- values - Values to bind to as parameters for the queries
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+In performance testing, FQL appears to have little or no overhead in ``WRITE`` only workloads, and a minor overhead in ``MIXED`` workload.
-The log for single CQL query includes the following attributes:
+Query information logged
+------------------------
-::
-
- query - CQL query text
- queryOptions - Options associated with the query invocation
- queryState - Timestamp state associated with the query invocation
- queryTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+The query log contains:
-Full query logging is backed up by ``BinLog``. BinLog is a quick and dirty binary log. Its goal is good enough performance, predictable footprint, simplicity in terms of implementation and configuration and most importantly minimal impact on producers of log records. Performance safety is accomplished by feeding items to the binary log using a weighted queue and dropping records if the binary log falls sufficiently far behind. Simplicity and good enough performance is achieved by using a single log writing thread as well as Chronicle Queue to handle writing the log, making it available for readers, as well as log rolling.
+- all queries invoked
+- approximate time they were invoked
+- any parameters necessary to bind wildcard values
+- all query options
-Weighted queue is a wrapper around any blocking queue that turns it into a blocking weighted queue. The queue will weigh each element being added and removed. Adding to the queue is blocked if adding would violate the weight bound. If an element weighs in at larger than the capacity of the queue then exactly one such element will be allowed into the queue at a time. If the weight of an object changes after it is added it could create issues. Checking weight should be cheap so memorize expensive to compute weights. If weight throws that can also result in leaked permits so it's always a good idea to memorize weight so it doesn't throw. In the interests of not writing unit tests for methods no one uses there is a lot of ``UnsupportedOperationException``. If you need them then add them and add proper unit tests to ``WeightedQueueTest``. "Good" tests. 100% coverage including exception paths and resource leaks.
+The logger writes single or batched CQL queries after they finish, so only successfully completed queries are logged. Failed or timed-out queries are not logged. Different data is logged, depending on the type of query.
+A single CQL query log entry contains:
-The FQL tracks information about store files:
+- query - CQL query text
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- queryTimeMillis - Approximate time in milliseconds since the epoch since the query was invoked
-- Store files as they are added and their storage impact. Delete them if over storage limit.
-- The files in the chronicle queue that have already rolled
-- The number of bytes in store files that have already rolled
+A batch CQL query log entry contains:
-FQL logger sequence is as follows:
+- queries - CQL text of the queries
+- queryOptions - Options associated with the query invocation
+- queryState - Timestamp state associated with the query invocation
+- batchTimeMillis - Approximate time in milliseconds since the epoch since the batch was invoked
+- type - The type of the batch
+- values - Values to bind to as parameters for the queries
-1. Start the consumer thread that writes log records. Can only be done once.
-2. Offer a record to the log. If the in memory queue is full the record will be dropped and offer will return false.
-3. Put a record into the log. If the in memory queue is full the putting thread will be blocked until there is space or it is interrupted.
-4. Clean up the buffers on thread exit, finalization will check again once this is no longer reachable ensuring there are no stragglers in the queue.
-5. Stop the consumer thread that writes log records. Can be called multiple times.
+Because FQL is backed by `Binlog`, the performance and footprint are predictable, with minimal impact on log record producers.
+Performance safety prevents the producers from overloading the log, using a weighted queue to drop records if the logging falls behind.
+Single-thread asynchronous writing produces the logs. Chronicle-Queue provides an easy method of rolling the logs.
-Next, we shall demonstrate full query logging with an example.
+Logging information logged
+--------------------------
+FQL also tracks information about the stored log files:
-Configuring Full Query Logging
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Stored log files that are added and their storage impact. Deletes them if over storage limit.
+- The log files in Chronicle-Queue that have already rolled
+- The number of bytes in the log files that have already rolled
-Full Query Logger default options are configured on a per node basis in ``cassandra.yaml`` with following configuration property.
+Logging sequence
+----------------
-::
+The logger follows a well-defined sequence of events:
- full_query_logging_options:
+1. The consumer thread that writes log records is started. This action can occur only once.
+2. The consumer thread offers a record to the log. If the in-memory queue is full, the record will be dropped and offer returns a `false` value.
+3. If accepted, the record is entered into the log. If the in-memory queue is full, the putting thread will be blocked until there is space or it is interrupted.
+4. The buffers are cleaned up at thread exit. Finalization will check again, to ensure there are no stragglers in the queue.
+5. The consumer thread is stopped. It can be called multiple times.
-As an example setup create a three node Cassandra 4.0 cluster. The ``nodetool status`` command lists the nodes in the cluster.
-
-::
+Using FQL
+---------
- [ec2-user@ip-10-0-2-238 ~]$ nodetool status
- Datacenter: us-east-1
- =====================
- Status=Up/Down
- |/ State=Normal/Leaving/Joining/Moving
- -- AddressLoad Tokens Owns (effective) Host ID Rack
- UN 10.0.1.115 442.42 KiB 25632.6% b64cb32a-b32a-46b4-9eeb-e123fa8fc287 us-east-1b
- UN 10.0.3.206 559.52 KiB 25631.9% 74863177-684b-45f4-99f7-d1006625dc9e us-east-1d
- UN 10.0.2.238 587.87 KiB 25635.5% 4dcdadd2-41f9-4f34-9892-1f20868b27c7 us-east-1c
+To use FQL, two actions must be completed. FQL must be configured using either the `cassandra.yaml` file or ``nodetool``, and logging must be enabled using ``nodetool enablefullquerylog``.
+Both actions are completed on a per-node basis.
+With either method, at a minimum, the path to the log directory must be specified.
+Full query logs are generated on each enabled node, so logs on each node will have that node's queries.
+Configuring FQL in cassandra.yaml
+---------------------------------
-In subsequent sub-sections we shall discuss enabling and configuring full query logging.
+The `cassandra.yaml` file can be used to configure FQL before enabling the feature with ``nodetool``.
-Setting the FQL Directory
-*************************
+The file includes the following options that can be uncommented for use:
-A dedicated directory path must be provided to write full query log data to when the full query log is enabled. The directory for FQL must exist, and have permissions set. The full query log will recursively delete the contents of this path at times. It is recommended not to place links in this directory to other sections of the filesystem. The ``full_query_log_dir`` property in ``cassandra.yaml`` is pre-configured.
+::
-::
-
- full_query_log_dir: /tmp/cassandrafullquerylog
-
-The ``log_dir`` option may be used to configure the FQL directory if the ``full_query_log_dir`` is not set.
-
-::
-
- full_query_logging_options:
+ # default options for full query logging - these can be overridden from command line
+ # when executing nodetool enablefullquerylog
+ #full_query_logging_options:
# log_dir:
+ # roll_cycle: HOURLY
+ # block: true
+ # max_queue_weight: 268435456 # 256 MiB
+ # max_log_size: 17179869184 # 16 GiB
+ ## archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled:
+ # archive_command:
+ # max_archive_retries: 10
-Create the FQL directory if it does not exist and set its permissions.
+log_dir
+^^^^^^^
-::
+To write logs, an existing directory must be set in ``log_dir``.
- sudo mkdir -p /tmp/cassandrafullquerylog
- sudo chmod -R 777 /tmp/cassandrafullquerylog
+The directory must have appropriate permissions set to allow reading, writing, and executing.
+Logging will recursively delete the directory contents as needed.
+Do not place links in this directory to other sections of the filesystem.
+For example, ``log_dir: /tmp/cassandrafullquerylog``.
-Setting the Roll Cycle
-**********************
+roll_cycle
+^^^^^^^^^^
-The ``roll_cycle`` option sets how often to roll FQL log segments so they can potentially be reclaimed. Supported values are ``MINUTELY``, ``HOURLY`` and ``DAILY``. Default setting is ``HOURLY``.
+The ``roll_cycle`` defines the frequency with which the log segments are rolled.
+Supported values are ``HOURLY`` (default), ``MINUTELY``, and ``DAILY``.
+For example: ``roll_cycle: DAILY``
-::
+block
+^^^^^
- roll_cycle: HOURLY
+The ``block`` option specifies whether FQL should block writing or drop log records if FQL falls behind. Supported boolean values are ``true`` (default) or ``false``.
+For example: ``block: false`` to drop records
-Setting Other Options
-*********************
+max_queue_weight
+^^^^^^^^^^^^^^^^
-The ``block`` option specifies whether the FQL should block if the FQL falls behind or should drop log records. Default value of ``block`` is ``true``. The ``max_queue_weight`` option sets the maximum weight of in memory queue for records waiting to be written to the file before blocking or dropping. The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The ``archive_command`` option sets the archive command to execute on rolled log files. The ``max_archive_retries`` option sets the max number of retries of failed archive commands.
+The ``max_queue_weight`` option sets the maximum weight of in-memory queue for records waiting to be written to the file before blocking or dropping. The option must be set to a positive value. The default value is 268435456, or 256 MiB.
+For example, to change the default: ``max_queue_weight: 134217728 # 128 MiB``
-::
+max_log_size
+^^^^^^^^^^^^
- # block: true
- # max_queue_weight: 268435456 # 256 MiB
- # max_log_size: 17179869184 # 16 GiB
- ## archive command is "/path/to/script.sh %path" where %path is replaced with the file
- being rolled:
- # archive_command:
- # max_archive_retries: 10
+The ``max_log_size`` option sets the maximum size of the rolled files to retain on disk before deleting the oldest file. The option must be set to a positive value. The default is 17179869184, or 16 GiB.
+For example, to change the default: ``max_log_size: 34359738368 # 32 GiB``
-The ``max_queue_weight`` must be > 0. Similarly ``max_log_size`` must be > 0. An example full query logging options is as follows.
+archive_command
+^^^^^^^^^^^^^^^
-::
+The ``archive_command`` option sets the user-defined archive script to execute on rolled log files.
Review comment:
Default is `""` which then maps to `org.apache.cassandra.utils.binlog.DeletingArchiver`
```
BinLogArchiver archiver = Strings.isNullOrEmpty(archiveCommand) ? new DeletingArchiver(maxLogSize) : new ExternalArchiver(archiveCommand, path, maxArchiveRetries);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org