You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by js...@apache.org on 2016/09/22 00:43:35 UTC
[trafficserver] branch master updated: TS-4715: total revamp of
admin guide's logging sections, and promotion to top-level chapter
This is an automated email from the ASF dual-hosted git repository.
jsime pushed a commit to branch master
in repository https://git-dual.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push:
new 6d2c6cb TS-4715: total revamp of admin guide's logging sections, and promotion to top-level chapter
6d2c6cb is described below
commit 6d2c6cbf0bd43cf21673cbc46f46bf51914068fe
Author: Jon Sime <js...@apache.org>
AuthorDate: Thu Sep 22 00:42:17 2016 +0000
TS-4715: total revamp of admin guide's logging sections, and promotion to top-level chapter
---
doc/admin-guide/files/logging.config.en.rst | 18 +-
doc/admin-guide/files/records.config.en.rst | 7 +-
doc/admin-guide/index.en.rst | 3 +-
doc/admin-guide/logging/cache-results.en.rst | 234 ++++
doc/admin-guide/logging/destinations.en.rst | 320 ++++++
doc/admin-guide/logging/examples.en.rst | 349 ++++++
.../log-builder.en.rst => logging/filters.en.rst} | 20 +-
doc/admin-guide/logging/formatting.en.rst | 757 +++++++++++++
.../{monitoring => }/logging/index.en.rst | 25 +-
.../rotation.en.rst} | 174 ++-
doc/admin-guide/logging/understanding.en.rst | 161 +++
doc/admin-guide/monitoring/error-messages.en.rst | 6 +-
doc/admin-guide/monitoring/index.en.rst | 5 +-
.../monitoring/logging/log-collation.en.rst | 142 ---
.../monitoring/logging/log-formats.en.rst | 1128 --------------------
doc/admin-guide/monitoring/logging/pipes.en.rst | 35 -
.../monitoring/logging/summary-logs.en.rst | 101 --
.../monitoring/logging/understanding.en.rst | 83 --
.../monitoring/third-party/logstash.en.rst | 12 +-
doc/static/images/admin/logcolat.jpg | Bin 112020 -> 0 bytes
20 files changed, 1947 insertions(+), 1633 deletions(-)
diff --git a/doc/admin-guide/files/logging.config.en.rst b/doc/admin-guide/files/logging.config.en.rst
index 328c829..df0b87b 100644
--- a/doc/admin-guide/files/logging.config.en.rst
+++ b/doc/admin-guide/files/logging.config.en.rst
@@ -92,7 +92,7 @@ Format Specification
The format specification provided as the required ``Format`` entry of the table
passed to the format function is a simple string, containing whatever mixture
of logging field variables and literal characters meet your needs. Logging
-fields are discussed in great detail in the :ref:`custom-logging-fields`
+fields are discussed in great detail in the :ref:`admin-logging-fields`
section.
Flexible enough to not only emulate the logging formats of most other proxy and
@@ -104,8 +104,8 @@ too simple format shown earlier, the following format string::
'%<cqtq>'
-Defines a format in which nothing but the value of the logging field
-:ref:`cqtq` is interpolated for each event's entry in the log. We could include
+Defines a format in which nothing but the value of the logging field :ref:`cqtq
+<cqtq>` is interpolated for each event's entry in the log. We could include
some literal characters in the log output by updating the format specification
as so::
@@ -137,7 +137,7 @@ header. Other stages of the event lifecycle have similar logging fields:
(proxy responses).
You will find a complete listing of the available fields in
-:ref:`custom-logging-fields`.
+:ref:`admin-logging-fields`.
Aggregation Interval
~~~~~~~~~~~~~~~~~~~~
@@ -149,7 +149,7 @@ interval do not behave like regular logs, with a single line for every event.
Instead, they emit a single line only every *interval*\ -seconds.
These types of logs are described in more detail in
-:ref:`admin-monitoring-logging-summary-logs`.
+:ref:`admin-logging-type-summary`.
Formats have no interval by default, and will generate event-based logs unless
given one.
@@ -187,7 +187,7 @@ Filter Fields
~~~~~~~~~~~~~
The log fields have already been discussed in the `Formats`_ section above. For
-a reference to the available log field names, see :ref:`custom-logging-fields`.
+a reference to the available log field names, see :ref:`admin-logging-fields`.
Unlike with the log format specification, you do not wrap the log field names
in any additional markup.
@@ -264,7 +264,7 @@ log.pipe(table)
There is no need to capture the return values of these functions. Which type of
logging output you choose depends largely on how you intend to process the logs
with other tools, and a discussion of the merits of each is covered elsewhere,
-in :ref:`admin-logging-binary-v-ascii`.
+in :ref:`admin-logging-ascii-v-binary`.
The following subsections cover the contents of the table which should be
passed when creating your logging object. Only ``Filename`` and ``Format`` are
@@ -298,8 +298,8 @@ Filters array of The optional list of filter objects which
CollationHosts array of If present, one or more strings specifying the
strings log collation hosts to which logs should be
delivered, each in the form of "<ip>:<port>".
- :ref:`admin-monitoring-logging-log-collation`
- for more information.
+ :ref:`admin-logging-collation` for more
+ information.
=================== =========== ===============================================
Enabling log rolling may be done globally in :file:`records.config`, or on a
diff --git a/doc/admin-guide/files/records.config.en.rst b/doc/admin-guide/files/records.config.en.rst
index 8298d91..5fa59a0 100644
--- a/doc/admin-guide/files/records.config.en.rst
+++ b/doc/admin-guide/files/records.config.en.rst
@@ -231,7 +231,7 @@ System Variables
.. ts:cv:: CONFIG proxy.config.syslog_facility STRING LOG_DAEMON
The facility used to record system log files. Refer to
- :ref:`admin-monitoring-logging-understanding` for more in-depth discussion
+ :ref:`admin-logging-understanding` for more in-depth discussion
of the contents and interpretations of log files.
.. ts:cv:: CONFIG proxy.config.cop.core_signal INT 0
@@ -2534,7 +2534,7 @@ Logging Configuration
``3`` Dull logging (errors and transactions).
======== ===================================================================
- Refer to :ref:`admin-monitoring-logging` for more information on event logging.
+ Refer to :ref:`admin-logging` for more information on event logging.
.. ts:cv:: CONFIG proxy.config.log.max_secs_per_buffer INT 5
:reloadable:
@@ -2633,7 +2633,8 @@ Logging Configuration
===== ======================================================================
For information on sending custom formats to the collation server,
- refer to :ref:`admin-monitoring-logging-formats` and :file:`logging.config`.
+ refer to :ref:`admin-logging-collating-custom-formats` and
+ :file:`logging.config`.
.. note::
diff --git a/doc/admin-guide/index.en.rst b/doc/admin-guide/index.en.rst
index fa39855..c2bf2eb 100644
--- a/doc/admin-guide/index.en.rst
+++ b/doc/admin-guide/index.en.rst
@@ -20,7 +20,7 @@
.. _admin-guide:
Administrator's Guide
-**********************
+*********************
Table of Contents:
@@ -34,6 +34,7 @@ Table of Contents:
security/index.en
storage/index.en
plugins/index.en
+ logging/index.en
monitoring/index.en
configuring-traffic-server.en
performance/index.en
diff --git a/doc/admin-guide/logging/cache-results.en.rst b/doc/admin-guide/logging/cache-results.en.rst
new file mode 100644
index 0000000..cafa797
--- /dev/null
+++ b/doc/admin-guide/logging/cache-results.en.rst
@@ -0,0 +1,234 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+.. _admin-logging-cache-results:
+
+Logging Cache Results
+*********************
+
+In addition to recording information about client and origin requests and
+responses, |TS| exposes detailed information about cache interactions for every
+request which passes through a node. This information is incredibly important
+to establishing an efficient caching configuration, though it is only present
+in your logging output if the :ref:`crc <crc>` logging field is used.
+
+Interpreting Cache Results
+==========================
+
+An immediate benefit to including cache result codes in your logging output is
+the ability to trace and diagnose specific problems. You may be able to tell
+without any help that a particular request takes much longer than it should,
+and HTTP response inspection reveals a very small (or possibly zero)
+value for the Age header, but you may still be left wondering why. If it's
+possible to correlate the problematic request(s) with access log entries, the
+cache result code may help shed significant light.
+
+Misconfigured headers from your origin may be resulting in clients aggressively
+sending ``no-cache`` requests, or cache objects may be going stale too rapidly
+and IMS (``If-Modified-Since``) requests are reaching back to the origin server
+more often than you anticipated. Perhaps this isn't a regular web browser
+request, but part of an application's API and the HTTP method being used isn't
+understood by |TS| and instead just gets passed blindly through to the origin.
+These and other situations can all be revealed pretty quickly by logging and
+inspecting |TS| cache results.
+
+Even more useful as an overall monitoring and optimization strategy is to log
+the cache results, and then produce reports through log aggregation and
+analysis tools. Such an approach can reveal larger trends and unearth problems
+or inefficiencies that don't necessarily result in outright client errors or
+bug reports from users.
+
+A high ratio of :ref:`crc-tcp-miss` to :ref:`crc-tcp-hit` events can indicate
+that your cache is sized too small and objects are being evicted too quickly
+for your |TS| cache to reduce traffic to your origin servers - largely
+defeating the purpose of having a caching layer at all. Here *too small* could
+be anything from simply not sized appropriately despite your hopes and
+intentions otherwise (hopefully because you're still fine-tuning your
+configuration based on monitoring and reporting results) to the difficult
+situation in which you operate very large sets of cacheable data that see too
+sparse access patterns to easily create a single cache policy.
+
+In the latter scenario, you may need to investigate tiered caching policies on
+subsets of your data to better tune the eviction rates, or partition data
+across different storage volumes to keep less-frequently, but very large
+objects (e.g. hours-long, archived training videos from years past) from
+suddenly evicting large numbers of more moderately visited, much smaller, but
+infrequently-changing objects (e.g. costly HTML pages of your semester course
+schedules generated by an unruly SQL query in your aging CMS).
+
+Large numbers of :ref:`crc-tcp-client-refresh` events for objects which are supposed to be
+served primarily (or perhaps almost exclusively) from your |TS| cache may
+indicate that something, possibly a misconfiguration of the caching headers in
+your origin server responses, is triggering clients to submit requests with
+``no-cache`` enabled.
+
+Almost any non-trivial occurrence of :ref:`crc-tcp-ref-fail-hit` requests is a
+signal that your origin servers are not responding to proxied requests from
+|TS| when they should. |TS| has found an object in its cache, but determining
+it to be stale attempted to refresh the object from an origin and was met with
+failure. The client still received an object back, but there was a chance it
+may have been outdated as a result of the origin failure.
+
+.. _admin-logging-crc:
+
+Cache Result Codes
+==================
+
+The following are all possible cache result codes you will currently encounter
+in |TS| logging output, and what each one means.
+
+.. _crc-tcp-hit:
+
+TCP_HIT
+-------
+
+A valid copy of the requested object was in the cache and Traffic Server sent
+the object to the client.
+
+.. _crc-tcp-miss:
+
+TCP_MISS
+--------
+
+The requested object was not in cache, so |TS| retrieved
+the object from the origin server (or a parent proxy) and sent it to
+the client.
+
+.. _crc-tcp-refresh-hit:
+
+TCP_REFRESH_HIT
+---------------
+
+The object was in the cache, but it was stale. |TS| made an
+``if-modified-since`` request to the origin server and the
+origin server sent a ``304`` not-modified response. Traffic
+Server sent the cached object to the client.
+
+.. _crc-tcp-ref-fail-hit:
+
+TCP_REF_FAIL_HIT
+----------------
+
+The object was in the cache but was stale. |TS| made an
+``if-modified-since`` request to the origin server but the server
+did not respond. |TS| sent the cached object to the
+client.
+
+.. _crc-tcp-refresh-miss:
+
+TCP_REFRESH_MISS
+----------------
+
+The object was in the cache but was stale. |TS| made an
+``if-modified-since`` request to the origin server and the server
+returned a new object. |TS| served the new object to the
+client.
+
+.. _crc-tcp-client-refresh:
+
+TCP_CLIENT_REFRESH
+------------------
+
+The client issued a request with a ``no-cache`` header. Traffic
+Server obtained the requested object from the origin server and sent
+a copy to the client. |TS| deleted the previous copy of
+the object from cache.
+
+.. _crc-tcp-ims-hit:
+
+TCP_IMS_HIT
+-----------
+
+The client issued an ``if-modified-since`` request and the object
+was in cache and fresher than the IMS date, or an
+``if-modified-since`` request to the origin server revealed the
+cached object was fresh. |TS| served the cached object to
+the client.
+
+.. _crc-tcp-ims-miss:
+
+TCP_IMS_MISS
+------------
+
+The client issued an
+``if-modified-since request`` and the object was either not in
+cache or was stale in cache. |TS| sent an
+``if-modified-since request`` to the origin server and received the
+new object. |TS| sent the updated object to the client.
+
+.. _crc-tcp-swapfail:
+
+TCP_SWAPFAIL
+------------
+
+The object was in the cache but could not be accessed. The client
+did not receive the object.
+
+.. _crc-err-client-abort:
+
+ERR_CLIENT_ABORT
+----------------
+
+The client disconnected before the complete object was sent.
+
+.. _crc-err-connect-fail:
+
+ERR_CONNECT_FAIL
+----------------
+
+|TS| could not reach the origin server.
+
+.. _crc-err-dns-fail:
+
+ERR_DNS_FAIL
+------------
+
+The Domain Name Server (DNS) could not resolve the origin server
+name, or no DNS could be reached.
+
+.. _crc-err-invalid-req:
+
+ERR_INVALID_REQ
+---------------
+
+The client HTTP request was invalid. (|TS| forwards
+requests with unknown methods to the origin server.)
+
+.. _crc-err-read-timeout:
+
+ERR_READ_TIMEOUT
+----------------
+
+The origin server did not respond to |TS|'s request within
+the timeout interval.
+
+.. _crc-err-proxy-denied:
+
+ERR_PROXY_DENIED
+----------------
+
+Client service was denied.
+
+.. _crc-err-unknown:
+
+ERR_UNKNOWN
+-----------
+
+The client connected, but subsequently disconnected without sending
+a request.
diff --git a/doc/admin-guide/logging/destinations.en.rst b/doc/admin-guide/logging/destinations.en.rst
new file mode 100644
index 0000000..11270a1
--- /dev/null
+++ b/doc/admin-guide/logging/destinations.en.rst
@@ -0,0 +1,320 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+.. _admin-logging-destinations:
+
+Log Destinations
+****************
+
+|TS| enables you to control where event log files are located, if and how they
+will be rotated, and how much space they can consume. The first of these topics
+is covered in this section, while the latter two will be discussed separately
+in :ref:`admin-logging-rotation-retention`.
+
+Two classes of destinations are provided by |TS| currently: local and remote.
+Local logging involves storing log data onto filesystems locally mounted on the
+same system as the |TS| processes themselves and are covered below in
+:ref:`admin-logging-destinations-local`, while remote logging options involving
+:manpage:`syslog` and built-in |TS| log collation, are covered below in
+:ref:`admin-logging-destinations-remote`.
+
+.. _admin-logging-destinations-local:
+
+Local Logging
+=============
+
+.. _admin-logging-directory:
+
+Log Directory Configuration
+---------------------------
+
+All local logging output (including incoming collation logs on a |TS| instance
+configured as a :ref:`admin-logging-collation-server`) are stored within a
+single base directory. Individual log file configurations may optionally append
+subdirectories to this base path. This location is adjusted with
+:ts:cv:`proxy.config.log.logfile_dir` in :file:`records.config`.
+
+This configuration may specify either an absolute path on the host (if it
+begins with ``/``) or a path relative to the |TS| installation directory (any
+setting which does not begin with ``/``).
+
+|TS| will need to be restarted, or you will need to run
+:option:`traffic_ctl config reload` for changes to the logging directory to
+take effect.
+
+Local Log Formats
+-----------------
+
+Local |TS| logs may be emitted in three different formats. The optimal format
+depends on how administrators intend to use the log data. The first two
+options, :ref:`admin-logging-ascii` and :ref:`admin-logging-binary` offer
+persistent storage of log data, which may be accessed and analyzed by other
+programs at any time (until the log file's configured rotation/retention
+policies, as discussed later in :ref:`admin-logging-rotation-retention`).
+
+The third option, :ref:`admin-logging-pipes` offers no persistent storage of
+log data, but rather a live stream of logged events which may be read and
+interpreted by external processes as they occur.
+
+.. _admin-logging-ascii:
+
+ASCII Log Files
+~~~~~~~~~~~~~~~
+
+ASCII |TS| logs are human-readable, plain-text files with output that is easily
+read directly and without the required aid of any additional processing or
+conversion tools. By default, log files in this format will have a ``.log``
+extension.
+
+.. _admin-logging-binary:
+
+Binary Log Files
+~~~~~~~~~~~~~~~~
+
+Binary log file output from |TS| avoids the conversion overhead of internal
+|TS| data structures to ASCII strings, but any use of these files by external
+programs (or just reading by a human) will first require the use of a converter
+application. Binary log files by default will have a ``.blog`` file extension.
+
+.. _admin-logging-pipes:
+
+Named Pipes
+~~~~~~~~~~~
+
+In addition to ASCII and binary file modes for custom log formats, |TS|
+can output log entries in ``ASCII_PIPE`` mode. This mode writes the log entries
+to a UNIX named pipe (a buffer in memory). Other processes may read from this
+named pipe using standard I/O functions.
+
+The advantage of this mode is that |TS| does not need to write the entries to
+disk, which frees disk space and bandwidth for other tasks. When the buffer is
+full, |TS| drops log entries and issues an error message indicating how many
+entries were dropped. Because |TS| only writes complete log entries to the
+pipe, only full records are dropped.
+
+Output to named pipes is always, as the mode's name implies, in ASCII format.
+There is no option for logging binary format log data to a named pipe.
+
+.. _admin-logging-ascii-v-binary:
+
+Deciding Between ASCII or Binary Output
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+|TS| offers both ASCII and binary output for log files because each offer
+advantages under different circumstances. The primary concerns and trade offs
+that should be considered are covered below. Many of the trade offs between
+formats will depend heavily on the specific formats you choose for your logs.
+To make an accurate determination on whether ASCII or binary logging is better
+for your systems, it is recommended that (with good system and performance
+monitoring, of course) that you test each format separately under real world
+traffic.
+
+The only blanket statement that can really be offered in good conscience is
+that ASCII logging *generally* offers a lower path of resistance as no
+additional conversion tools will be necessary.
+
+Disk Space
+^^^^^^^^^^
+
+ASCII logs tend to consume more disk space than their binary counterparts.
+Many numeric fields (e.g. content lengths, HTTP status codes, request and
+response times, and so on) as well as string representation of IPv4 and IPv6
+addresses will consume more bytes than their binary formats. There are
+exceptions (a field containing just the value ``0`` will use a single byte in
+an ASCII log, but four bytes in a binary log), so a guarantee cannot be made,
+but the general tendency for typical log line formats is to consume slightly
+more space in ASCII.
+
+CPU Overhead
+^^^^^^^^^^^^
+
+Emitting ASCII format logs does incur some additional processing as the
+internal |TS| data structures for relevant transaction details need to be
+converted into ASCII strings. While this is usually negligible overhead for
+most installations, you may wish to compare the performance overhead between
+emitting ASCII or binary log data if you are very concerned with |TS| runtime
+performance. By using the binary log format, you may gain a very slight amount
+of proxy performance, at the cost of having to invoke an intermediary converter
+application every time you wish to view or process the log data.
+
+External Programs
+^^^^^^^^^^^^^^^^^
+
+As mentioned above, any use of the log data by other programs will require the
+addition of a converter application should you opt for the binary format. If
+you are frequently ingesting the log data elsewhere, you may not wish to have
+the time and processing cost of this additional step every time.
+
+If the external program is ingesting the logs continuously, you may wish to use
+a named pipe output from |TS| instead, which is always in ASCII format, but
+doesn't have the potentially increased storage needs as there is no persistent
+storage of the log data involved (at least not by |TS| - the application
+ingesting the data is probably storing its own results somewhere). It also
+avoides unnecessary disk I/O operations if you only care about the final,
+analyzed version of the log data and have no permanent use for the intermediate
+(and raw) output from |TS|.
+
+Alternatively, if you wish to hyper-optimize your |TS| runtime performance and
+are only ingesting the log data with an external application on a batched
+schedule, you might consider logging from |TS| using the binary format, then
+establishing an externally scheduled one-time conversion of the log data to a
+more easily ingested ASCII format into separate file(s). Coordination of this
+conversion with the |TS| log rotations would be your responsibility.
+
+.. _admin-logging-destinations-remote:
+
+Remote Logging
+==============
+
+|TS| provides for remote log-shipping functionality, which may be used in
+addition to or instead of local log storage. This section covers the current
+options available.
+
+.. _admin-logging-syslog:
+
+Syslog
+------
+
+At this time, |TS| supports sending log data to :manpage:`syslog` only for the
+system and emergency logs. Sending custom event or transaction error logs to
+syslog is not directly supported. You may use external log aggregation tools,
+such as Logstash, to accomplish this by having them handle the ingestion of
+|TS| local log files and forwarding to whatever receivers you wish.
+
+.. _admin-logging-collation:
+
+Log Collation
+-------------
+
+|TS| offers remote log shipping natively through the log collation feature,
+which allows one or more |TS| instances handling regular traffic to transmit
+their log data to one or more |TS| instances acting as collation servers.
+
+This allows you to centralize your |TS| logging for (potentially) easier
+analysis and reporting in environments with many |TS| instances. Collation
+servers may be members of a |TS| cluster or entirely independent servers
+running a stripped down configuration aimed at log collation only (and omitting
+any configuration for actual traffic proxying or caching).
+
+When a |TS| node generates a buffer of event log entries, it first determines
+if it is the collation server or a collation client. The collation server node
+writes all log buffers to its local disk (as per its :file:`logging.config`
+configuration), just as it would if log collation was not enabled. The
+collation client nodes prepare their log buffers for transfer across the
+network and send the buffers to the configured log collation server.
+
+If log clients cannot contact their log collation server, then they write their
+log buffers to their local disks, into orphan log files. Orphaned log files
+require manual collation.
+
+.. important::
+
+ Log collation can have an impact on network performance. Because all nodes
+ are forwarding their log data buffers to the single collation server, a
+ bottleneck can occur. In addition, collated log files contain timestamp
+ information for each entry, but entries in the files do not appear in strict
+ chronological order. You may want to sort collated log files before doing
+ analysis.
+
+.. _admin-logging-collation-client:
+
+Collation Client
+~~~~~~~~~~~~~~~~
+
+To configure a |TS| node to be a collation client, follow the steps below.
+
+#. In the :file:`records.config` file, edit the following variables:
+
+ - :ts:cv:`proxy.local.log.collation_mode`: ``2`` to configure this node as
+ log collation client and send standard formatted log entries to the
+ collation server. For custom log entries, see :file:`logging.config`.
+ - :ts:cv:`proxy.config.log.collation_host`
+ - :ts:cv:`proxy.config.log.collation_port`
+ - :ts:cv:`proxy.config.log.collation_secret`
+ - :ts:cv:`proxy.config.log.collation_host_tagged`
+ - :ts:cv:`proxy.config.log.max_space_mb_for_orphan_logs`
+
+#. Run the command :option:`traffic_ctl config reload` to apply the configuration
+ changes.
+
+.. note::
+
+ If you modify the collation port or secret after connections between the
+ collation server and collation clients have been established, you must
+ restart |TS| on all nodes.
+
+.. _admin-logging-collation-server:
+
+Collation Server
+~~~~~~~~~~~~~~~~
+
+To configure a |TS| node to be a collation server, perform the following
+configuration adjustments in :file:`records.config`:
+
+#. Set :ts:cv:`proxy.local.log.collation_mode` to ``1`` to indicate this node
+ will be a server. ::
+
+ CONFIG proxy.local.log.collation_mode INT 1
+
+#. Configure the port on which the server will listen to incoming collation
+ transfers from clients, using :ts:cv:`proxy.config.log.collation_port`. If
+ omitted, this defaults to port ``8085``. ::
+
+ CONFIG proxy.config.log.collation_port INT 8085
+
+#. Configure the shared secret used by collation clients to authenticate their
+ sessions, using :ts:cv:`proxy.config.log.collation_secret`. ::
+
+ CONFIG proxy.config.log.collation_secret STRING seekrit
+
+#. Run the command :option:`traffic_ctl config reload` to apply the configuration
+ changes.
+
+.. note::
+
+ If you modify the collation port or secret after connections between the
+ collation server and collation clients have been established, you must
+ restart |TS| on all nodes.
+
+.. _admin-logging-collating-custom-formats:
+
+Collating Custom Logs
+~~~~~~~~~~~~~~~~~~~~~
+
+If you use custom event log files, then you must edit :file:`logging.config`,
+in addition to configuring a collation server and collation clients.
+
+To collate custom event log files:
+
+#. On each collation client, edit :file:`logging.config` and add the
+ ``CollationHosts`` attribute to the relevant logs. For example, adding two
+ collation hosts to an ASCII log that uses the Squid format would look like:
+
+ .. code:: lua
+
+ log.ascii {
+ Format = squid,
+ Filename = 'squid',
+ CollationHosts = { '192.168.1.100:4567', '192.168.1.101:4567' }
+ }
+
+#. Run the command :option:`traffic_ctl config reload` to restart |TS| on the
+ local node or :option:`traffic_ctl config reload` to restart |TS| on all
+ the nodes in a cluster.
+
diff --git a/doc/admin-guide/logging/examples.en.rst b/doc/admin-guide/logging/examples.en.rst
new file mode 100644
index 0000000..9e515bb
--- /dev/null
+++ b/doc/admin-guide/logging/examples.en.rst
@@ -0,0 +1,349 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+.. _admin-logging-examples:
+
+Configuration Examples
+**********************
+
+This section provides examples for a wide range of logging scenarios. While not
+exhaustive of all possibilities (|TS| logging is quite flexible), these entries
+should hopefully get most administrators headed onto the right path.
+
+Unless otherwise noted, the example configurations here are to be applied in
+:file:`logging.config`.
+
+Online Event Log Builder
+========================
+
+If you need any assistance building your event log, you can try out our
+`online log builder <http://trafficserver.apache.org/logbuilder/>`_. This is a
+work in progress, so any comments, critique or suggestions are most welcome.
+
+Emulating Other HTTP Server Formats
+===================================
+
+.. _admin-logging-examples-netscape:
+
+Netscape Common
+---------------
+
+The following figure shows a sample log entry in a Netscape Common log file.
+
+.. figure:: /static/images/admin/netscape_common_format.jpg
+ :align: center
+ :alt: Sample Netscape Common log entry
+
+The numbered sections correspond to the following log fields in |TS|:
+
+=== ====== ====================================================================
+No. Field Description
+=== ====== ====================================================================
+1 chi The IP address of the client's host machine.
+2 -- This hyphen (``-``) is always present in Netscape log entries.
+3 caun The authenticated client username. A hyphen (``-``) means no
+ authentication was required.
+4 cqtn The date and time of the client request, enclosed in brackets.
+5 cqtx The request line, enclosed in quotes.
+6 pssc The proxy response status code (HTTP reply code).
+7 pscl The length of the |TS| response to the client in bytes.
+=== ====== ====================================================================
+
+To recreate this as a log format in :file:`logging.config` you would define the
+following format object:
+
+.. code:: lua
+
+ common = format {
+ Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
+ }
+
+.. _admin-logging-examples-extended:
+
+Netscape Extended
+-----------------
+
+The following figure shows a sample log entry in a Netscape Extended log file.
+
+.. figure:: /static/images/admin/netscape_extended_format.jpg
+ :align: center
+ :alt: Sample Netscape Extended log entry
+
+In addition to fields 1-7 from the Netscape Common log format, the Extended
+format adds the following fields:
+
+=== ===== =====================================================================
+No. Field Description
+=== ===== =====================================================================
+8 sssc The origin server response status code.
+9 sshl The server response transfer length; the body length in the
+ origin server response to |TS|, in bytes.
+10 cqcl The client request transfer length; the body length in the
+ client request to |TS|, in bytes.
+11 pqcl The proxy request transfer length; the body length in the |TS|
+ request to the origin server.
+12 cqhl The client request header length; the header length in the
+ client request to |TS|.
+13 pshl The proxy response header length; the header length in the
+ |TS| response to the client.
+14 pqhl The proxy request header length; the header length in |TS|
+ request to the origin server.
+15 sshl The server response header length; the header length in the
+ origin server response to |TS|.
+16 tts The time |TS| spent processing the client request; the number
+ of seconds between the time that the client established the
+ connection with |TS| and the time that |TS| sent the last byte
+ of the response back to the client.
+=== ===== =====================================================================
+
+To recreate this as a log format in :file:`logging.config` you would define the
+following format object:
+
+.. code:: lua
+
+ extended = format {
+ Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl> %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts>'
+ }
+
+.. _admin-logging-examples-extended2:
+
+Netscape Extended-2
+-------------------
+
+The following figure shows a sample log entry in a Netscape Extended2 log file.
+
+.. figure:: /static/images/admin/netscape_extended2_format.jpg
+ :align: center
+ :alt: Sample Netscape Extended-2 log entry
+
+In addition to fields 1-16 from the Netscape Common and Netscape Extended log
+formats above, the Extended-2 format adds the following fields:
+
+=== ====== ===============================================================
+No. Field Description
+=== ====== ===============================================================
+17 phr The proxy hierarchy route; the route |TS| used to retrieve
+ the object.
+18 cfsc The client finish status code: ``FIN`` if the client request
+ completed successfully or ``INTR`` if the client request was
+ interrupted.
+19 pfsc The proxy finish status code: ``FIN`` if the |TS| request to
+ the origin server completed successfully or ``INTR`` if the
+ request was interrupted.
+20 crc The cache result code; how the |TS| cache responded to the
+ request: ``HIT``, ``MISS``, and so on. Cache result codes are
+ listed in :ref:`admin-logging-crc`.
+=== ====== ===============================================================
+
+To recreate this as a log format in :file:`logging.config` you would define the
+following format object:
+
+.. code:: lua
+
+ extended2 = format {
+ Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl> %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts> %<phr> %<cfsc> %<pfsc> %<crc>'
+ }
+
+.. _admin-logging-examples-squid:
+
+Squid
+-----
+
+The following figure shows a sample log entry in a Squid log file.
+
+.. figure:: /static/images/admin/squid_format.jpg
+ :align: center
+ :alt: Sample Squid log entry
+
+The numbered sections correspond to the following log fields in |TS|:
+
+=== ======== ==================================================================
+No. Field Description
+=== ======== ==================================================================
+1 cqtq The client request timestamp in Squid format. The time of the
+ client request in seconds since January 1, 1970 UTC (with
+ millisecond resolution).
+2 ttms The time |TS| spent processing the client request. The number
+ of milliseconds between the time the client established the
+ connection with |TS| and the time |TS| sent the last byte of
+ the response back to the client.
+3 chi The IP address of the client\u2019s host machine.
+4 crc/pssc The cache result code; how the cache responded to the request:
+ ``HIT``, ``MISS``, and so on. Cache result codes are described
+ in :ref:`admin-logging-cache-results`. The
+ proxy response status code (HTTP response status code from
+ |TS| to client).
+5 psql The length of the |TS| response to the client in bytes,
+ including headers and content.
+6 cqhm The client request method: ``GET``, ``POST``, and so on.
+7 cauc The client request canonical URL; blanks and other characters
+ that might not be parsed by log analysis tools are replaced by
+ escape sequences. The escape sequence is a percentage sign
+ followed by the ASCII code number of the replaced character in
+ hex.
+8 caun The username of the authenticated client. A hyphen (``-``)
+ means that no authentication was required.
+9 phr/pqsn The proxy hierarchy route. The route |TS| used to retrieve the
+ object.
+10 psct The proxy response content type. The object content type taken
+ from the |TS| response header.
+=== ======== ==================================================================
+
+To recreate this as a log format in :file:`logging.config` you would define the
+following format object:
+
+.. code:: lua
+
+ squid = format {
+ Format = '%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>'
+ }
+
+Separating Logs by Origin
+=========================
+
+The default :file:`log_hosts.config` file is located in the |TS| ``config``
+directory. To record HTTP transactions for different origin servers in separate
+log files, you must specify the hostname of each origin server on a separate
+line in :file:`log_hosts.config`. For example, if you specify the keyword
+``sports``, then |TS| records all HTTP transactions from ``sports.yahoo.com``
+and ``www.foxsports.com`` in a log file called ``squid-sports.log`` (if the
+Squid format is enabled).
+
+.. important::
+
+ If |TS| is clustered and you enable log file collation, then you should use
+ the same :file:`log_hosts.config` file on every |TS| node in the cluster.
+
+To edit the log hosts list:
+
+#. Enter the hostname of each origin server on a separate line in
+ :file:`log_hosts.config`. ::
+
+ webserver1
+ webserver2
+ webserver3
+
+#. Run the command :option:`traffic_ctl config reload` to apply the changes.
+
+Hourly Rotated Squid Proxy Logs
+===============================
+
+The following example demonstrates the creation of a Squid-compatible log
+format, which is then applied to a log object containing an hourly rotation
+policy.
+
+.. code:: lua
+
+ squid = format {
+ Format = '%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>'
+ }
+
+ log.ascii {
+ Format = squid,
+ Filename = 'squid',
+ RollingEnabled = 1,
+ RollingIntervalSec = 3600,
+ RollingOffsetHr = 0
+ }
+
+Summarizing Number of Requests and Total Bytes Sent Every 10 Seconds
+====================================================================
+
+The following example format generates one entry every 10 seconds. Each entry
+contains the timestamp of the last entry of the interval, a count of the number
+of entries seen within that 10-second interval, and the sum of all bytes sent
+to clients:
+
+.. code:: lua
+
+ mysummary = format {
+ Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>",
+ Interval = "10"
+ }
+
+Dual Output to Compact Binary Logs and ASCII Pipes
+==================================================
+
+This example demonstrates logging the same event data to multiple locations, in
+a hypothetical scenario where we may wish to keep a compact form of our logs
+available for archival purposes, while performing live log analysis on a stream
+of the event data.
+
+.. code:: lua
+
+ ourformat = format {
+ Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
+ }
+
+ log.binary {
+ Format = ourformat,
+ Filename = 'archived_events'
+ }
+ log.pipe {
+ Format = ourformat,
+ Filename = 'streaming_log'
+ }
+
+Filtering Events to ASCII Pipe for Alerting
+===========================================
+
+This example illustrates a situation in which our |TS| cache contains *canary*
+objects, which upon their access we want an external alerting system to fire
+off all sorts of alarms. To accomplish this, we demonstrate the use of a filter
+object that matches events against these particular canaries and emits log data
+for them to a UNIX pipe that the alerting software can constantly read from.
+
+.. code:: lua
+
+ canaryformat = format {
+ Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
+ }
+
+ canaryfilter = filter.accept('cqup MATCH "/nightmare/scenario/dont/touch"')
+
+ log.pipe {
+ Format = canaryformat,
+ Filters = [ canaryfilter ],
+ Filename = 'alerting_canaries'
+ }
+
+Summarizing Origin Responses by Hour
+====================================
+
+This example demonstrates a simple use of aggregation operators to produce an
+hourly event line reporting on the total number of requests made to origin
+servers (where we assume that any cache result code without the string ``HIT``
+in it signals origin access), as well as the average time it took to fulfill
+the request to clients during that hour.
+
+.. code:: lua
+
+ originrepformat = format {
+ Format = '%<FIRST(cqtq)> %<COUNT(*)> %<AVERAGE(ttms)>',
+ Interval = 3600
+ }
+
+ originfilter = filter.reject('crc CONTAINS "HIT"')
+
+ log.ascii {
+ Format = originrepformat,
+ Filters = [ originfilter ],
+ Filename = 'origin_access_summary'
+ }
+
diff --git a/doc/admin-guide/monitoring/logging/log-builder.en.rst b/doc/admin-guide/logging/filters.en.rst
similarity index 54%
rename from doc/admin-guide/monitoring/logging/log-builder.en.rst
rename to doc/admin-guide/logging/filters.en.rst
index 1b80757..e0c3bfb 100644
--- a/doc/admin-guide/monitoring/logging/log-builder.en.rst
+++ b/doc/admin-guide/logging/filters.en.rst
@@ -15,12 +15,20 @@
specific language governing permissions and limitations
under the License.
-.. include:: ../../../common.defs
+.. include:: ../../common.defs
-Event Log Builder
-*****************
+.. _admin-logging-filters:
-If you need any assistance building your event log, you can try out our
-`online log builder <http://trafficserver.apache.org/logbuilder/>`_. This is a
-work in progress, so any comments, critique or suggestions are most welcome.
+Log Filters
+***********
+Filters, configured in :file:`logging.config` allow you to create rules which
+may be applied to log definitions, limiting the entries included in the log
+output. This may be useful if your |TS| nodes receive many events which you
+have no need to log or analyze, or you wish to establish separate logs with
+their own rotation policies to more rapidly perform log analysis for a subset
+of events.
+
+Configuration options are covered in detail in the
+:ref:`admin-custom-logs-filters` section of :file:`logging.config`, so this
+page is currently left as a reference to the feature's existence.
diff --git a/doc/admin-guide/logging/formatting.en.rst b/doc/admin-guide/logging/formatting.en.rst
new file mode 100644
index 0000000..346db45
--- /dev/null
+++ b/doc/admin-guide/logging/formatting.en.rst
@@ -0,0 +1,757 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+.. _admin-logging-formats:
+
+Formatting Log Files
+********************
+
+This section covers the creation of logging formats. All but a few logging
+output related settings in |TS| are performed in :file:`logging.config` and
+consulting the documentation for that file is recommended in addition to this
+section. Any configurations or settings performed outside that file will be
+clearly noted.
+
+.. _admin-logging-defining-format:
+
+Defining Formats
+================
+
+Logging formats in |TS| are defined by editing :file:`logging.config`
+and adding new format entries for each format you wish to define. The syntax is
+fairly simple: every format must contain a ``Format`` attribute, which is the
+string defining the contents of each line in the log, and may also contain an
+optional ``Interval`` attribute defining the log aggregation interval for
+any logs which use the format (see :ref:`admin-logging-type-summary` for more
+information).
+
+The return value from the ``format`` function is the log format object which
+may then be supplied to the appropriate ``log.*`` functions that define your
+logging destinations.
+
+A very simple exampe, which contains only the timestamp of when the event began
+and the canonical URL of the request, would look like:
+
+.. code:: lua
+
+ myformat = format {
+ Format = "%<cqtq> %<cauc>"
+ }
+
+You may include as many custom field codes as you wish. The full list of codes
+available can be found in :ref:`admin-logging-fields`. You may also include
+any literal characters in your format. For example, if we wished to separate
+the timestamp and canonical URL in our customer format above with a slash
+instead of a space, or even a slash surrounded by spaces, we could do so by
+just adding the desired characters to the format string:
+
+.. code:: lua
+
+ myformat = format {
+ Format = "%<cqtq> / %<cauc>"
+ }
+
+You may define as many custom formats as you wish. To apply changes to custom
+formats, you will need to run the command :option:`traffic_ctl config reload`
+after saving your changes to :file:`logging.config`.
+
+.. _admin-logging-fields:
+
+Log Fields
+==========
+
+The following sections detail all the available |TS| logging fields, broken
+down into the following broad categories for (hopefully) easier reference:
+
+- :ref:`admin-logging-fields-auth`
+- :ref:`admin-logging-fields-cache`
+- :ref:`admin-logging-fields-txn`
+- :ref:`admin-logging-fields-content-type`
+- :ref:`admin-logging-fields-hierarchy`
+- :ref:`admin-logging-fields-headers`
+- :ref:`admin-logging-fields-methods`
+- :ref:`admin-logging-fields-ids`
+- :ref:`admin-logging-fields-lengths`
+- :ref:`admin-logging-fields-collation`
+- :ref:`admin-logging-fields-network`
+- :ref:`admin-logging-fields-plugin`
+- :ref:`admin-logging-fields-proto`
+- :ref:`admin-logging-fields-request`
+- :ref:`admin-logging-fields-ssl`
+- :ref:`admin-logging-fields-status`
+- :ref:`admin-logging-fields-tcp`
+- :ref:`admin-logging-fields-time`
+- :ref:`admin-logging-fields-urls`
+
+Individual log fields are used within a log format string by enclosing them in
+angle brackets and prefixing with a percent sign. For example, to use the log
+field cqhl_ (the length in bytes of the client request headers), you
+would do the following::
+
+ Format = '%<cqhl>'
+
+Literal characters may be used, but they must be outside the log fields'
+placeholders, as so::
+
+ Format = 'Client Header Length (bytes): %<cqhl>'
+
+You may combine many fields into a single format string (logs wouldn't be very
+useful if you couldn't). Some fields do require a little extra treatment, which
+is noted clearly in their descriptions below. This affects, primarily, those
+fields which provide access to HTTP header values as you need to specify which
+header's value you wish to appear in the log data. For these, the header name
+is noted inside the angle brackets, before the log field name, and are
+enclosed within a curly braces pair. For example, to include the value of the
+Age header from an origin server response you would do::
+
+ Format = '%<{Age}ssh>'
+
+.. _admin-logging-fields-auth:
+
+Authentication
+~~~~~~~~~~~~~~
+
+.. _caun:
+
+These log fields provide access to various details of a client or proxy's
+means of request authentication to their destination (whether it be the client
+request to a proxy server, or the proxy server's request to an origin).
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+caun Client Request Authentication User name as a result of the RFC931/ident
+ lookup for the client-provided name.
+===== ============== ==========================================================
+
+.. _admin-logging-fields-cache:
+
+Cache Details
+~~~~~~~~~~~~~
+
+.. _cluc:
+.. _crc:
+.. _chm:
+.. _cwr:
+.. _cwtr:
+
+These log fields reveal details of the |TS| proxy interaction with its own
+cache while attempting to service incoming client requests.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cluc Client Request Cache Lookup URL, also known as the :term:`cache key`,
+ which is the canonicalized version of the client request
+ URL.
+crc Proxy Cache Cache Result Code. The result of |TS| attempting to obtain
+ the object from cache; :ref:`admin-logging-cache-results`.
+chm Proxy Cache Cache Hit-Miss status. Specifies the level of cache from
+ which this request was served by |TS|. Currently supports
+ only RAM (``HIT_RAM``) vs disk (``HIT_DISK``).
+cwr Proxy Cache Cache Write Result. Specifies the result of attempting to
+ write to cache: not relevant (``-``), no cache write
+ (``WL_MISS``), write interrupted (``INTR``), error while
+ writing (``ERR``), or cache write successful (``FIN``).
+cwt Proxy Cache Cache Write Transform Result.
+===== ============== ==========================================================
+
+.. _admin-logging-fields-txn:
+
+Connections and Transactions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _sca:
+.. _sstc:
+
+The following log fields are used to list various details of connections and
+transactions between |TS| proxies and origin servers.
+
+===== ====== ==================================================================
+Field Source Description
+===== ====== ==================================================================
+sca Proxy Number of attempts within the current transaction by |TS|
+ in connecting to the origin server.
+sstc Proxy Number of transactions between the |TS| proxy and the origin
+ server from a single session. Any value greater than zero
+ indicates connection reuse.
+===== ====== ==================================================================
+
+.. _admin-logging-fields-content-type:
+
+Content Types
+~~~~~~~~~~~~~
+
+.. _psct:
+
+Log fields used to reveal the HTTP content types in effect for transactions.
+
+===== ====================== ==================================================
+Field Source Description
+===== ====================== ==================================================
+psct Origin Server Response Content type of the document obtained by |TS| from
+ the origin server response.
+===== ====================== ==================================================
+
+.. _admin-logging-fields-hierarchy:
+
+Hierarchical Proxies
+~~~~~~~~~~~~~~~~~~~~
+
+.. _phr:
+
+The log fields detail aspects of transactions involving hierarchical caches.
+
+===== ====== ==================================================================
+Field Source Description
+===== ====== ==================================================================
+phr Proxy Proxy Hierarchy Route. Specifies the route through configured
+ hierarchical caches used to retrieve the object.
+===== ====== ==================================================================
+
+.. _admin-logging-fields-headers:
+
+HTTP Headers
+~~~~~~~~~~~~
+
+.. _cqh:
+.. _pqh:
+.. _psh:
+.. _ssh:
+.. _cssh:
+.. _ecqh:
+.. _epqh:
+.. _epsh:
+.. _essh:
+.. _ecssh:
+
+The following log tables provide access to the values of specified HTTP headers
+from each phase of the transaction lifecycle. Unlike many of the other log
+fields, these require a little extra notation in the log format string, so that
+|TS| knows the individual HTTP header from which you aim to extract a value for
+the log entry.
+
+This is done by specifying the name of the HTTP header in curly braces, just
+prior to the log field's name, as so::
+
+ Format = '%<{User-agent}cqh>'
+
+The above would insert the User Agent string from the client request headers
+into your log entry (or a blank string if no such header was present, or it did
+not contain a value).
+
+===== ====================== ==================================================
+Field Source Description
+===== ====================== ==================================================
+cqh Client Request Logs the value of the named header from the
+ client's request to the |TS| proxy.
+pqh Proxy Request Logs the value of the named header from the |TS|
+ proxy's request to the origin server.
+psh Proxy Response Logs the value of the named header from the |TS|
+ proxy's response to the client.
+ssh Origin Response Logs the value of the named header from the origin
+ server's response to the proxy.
+cssh Cached Origin Response Logs the value of the named header from the
+ *cached* origin server response.
+===== ====================== ==================================================
+
+Each of these also includes a URI-encoded variant, which replaces various
+characters in the string with entity encodings - rendering them safe for use in
+URL path components or query parameters. The variants' names follow the pattern
+of the origin field named prefixed with ``e``, as shown here:
+
+============== ===================
+Original Field URL-Encoded Variant
+============== ===================
+cqh ecqh
+pqh epqh
+psh epsh
+ssh essh
+cssh ecssh
+============== ===================
+
+.. _admin-logging-fields-methods:
+
+HTTP Methods
+~~~~~~~~~~~~
+
+.. _cqhm:
+
+These fields are used to log information about the HTTP methods/verbs used by
+requests.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cqhm Client Request HTTP method used in the client request to the |TS| proxy
+ (e.g. ``GET``, ``POST``, etc.).
+===== ============== ==========================================================
+
+.. _admin-logging-fields-ids:
+
+Identifiers
+~~~~~~~~~~~
+
+.. _crid:
+.. _cruuid:
+.. _puuid:
+
+Logging fields used to obtain various unique identifiers for transactions or
+objects.
+
+====== ============== =========================================================
+Field Source Description
+====== ============== =========================================================
+crid Client Request Sequence number of the current client request. Resets to
+ ``0`` on every |TS| restart.
+cruuid Client Request UUID of the current client request; generated by
+ concatenating the puuid_ and crid_ field values.
+puuid Proxy Server UUID for the currently running :program:`traffic_server`
+ process. Regenerated on every |TS| startup.
+====== ============== =========================================================
+
+.. _admin-logging-fields-lengths:
+
+Lengths and Sizes
+~~~~~~~~~~~~~~~~~
+
+.. _cqcl:
+.. _cqhl:
+.. _cqql:
+.. _csscl:
+.. _csshl:
+.. _cssql:
+.. _fsiz:
+.. _pqcl:
+.. _pqhl:
+.. _pqql:
+.. _pscl:
+.. _pshl:
+.. _psql:
+.. _sscl:
+.. _sshl:
+.. _ssql:
+
+These log fields are used to obtain various lengths and sizes of transaction
+components (headers, content bodies, etc.) between clients, proxies, and
+origins. Unless otherwise noted, all lengths are in bytes.
+
+===== ====================== ==================================================
+Field Source Description
+===== ====================== ==================================================
+cqcl Client Request Client request content length, in bytes.
+cqhl Client Request Client request header length, in bytes.
+cqql Client Request Client request header and content length combined,
+ in bytes.
+csscl Cached Origin Response Content body length from cached origin response.
+csshl Cached Origin Response Header length from cached origin response.
+cssql Cached Origin Response Content and header length from cached origin
+ response.
+fsiz Origin Size of the file as seen by the origin server.
+pqcl Proxy Request Content body length of the |TS| proxy request to
+ the origin server.
+pqhl Proxy Request Header length of the |TS| proxy request to the
+ origin server.
+pqql Proxy Request Content body and header length combined, of the
+ |TS| request to the origin server.
+pscl Proxy Response Content body length of the |TS| proxy response.
+pshl Proxy Response Header length of the |TS| response to client.
+psql Proxy Response Content body and header length combined of the
+ |TS| response to client.
+sscl Origin Response Content body length of the origin server response
+ to |TS|.
+sshl Origin Response Header length of the origin server response.
+ssql Origin Response Content body and header length combined of the
+ origin server response to |TS|.
+===== ====================== ==================================================
+
+.. _admin-logging-fields-collation:
+
+Log Collation
+~~~~~~~~~~~~~
+
+.. _phn:
+.. _phi:
+
+Logging fields related to :ref:`admin-logging-collation`.
+
+===== ====== ==================================================================
+Field Source Description
+===== ====== ==================================================================
+phn Proxy Hostname of the |TS| node which generated the collated log entry.
+phi Proxy IP of the |TS| node which generated the collated log entry.
+===== ====== ==================================================================
+
+.. _admin-logging-fields-network:
+
+Network Addresses, Ports, and Interfaces
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _chi:
+.. _chih:
+.. _hii:
+.. _hiih:
+.. _chp:
+.. _php:
+.. _pqsi:
+.. _pqsp:
+.. _pqsn:
+.. _shi:
+.. _shn:
+
+The following log fields are used to log details of the network (IP) addresses,
+incoming/outgoing ports, and network interfaces used during transactions.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+chi Client IP address of the client's host.
+chih Client IP address of the client's host, in hexadecimal.
+hii Proxy IP address for the proxy's incoming interface (to which
+ the client connected).
+hiih Proxy IP address for the proxy's incoming interface (to which
+ the client connected), in hexadecimal.
+chp Client Port number of the client's host.
+php Proxy Response TCP port number from which |TS| serviced the request.
+pqsi Proxy Request IP address from which |TS| issued the proxy request to the
+ origin server. Cache hits will result in a value of ``0``.
+pqsp Proxy Request Port number from which |TS| issued the proxy request to
+ the origin server. Cache hits will yield a value of ``0``.
+pqsn Proxy Request Host name of the interface from which |TS| issues the
+ proxy request to the origin server.
+shi Origin Server IP address resolved via DNS by |TS| for the origin server.
+ For hosts with multiple IP addresses, the address used by
+ |TS| for the connection will be reported. See note below
+ regarding misleading values from cached documents.
+shn Origin Server Host name of the origin server.
+===== ============== ==========================================================
+
+.. note::
+
+ This can be misleading for cached documents. For example: if the first
+ request was a cache miss and came from *IP1* for server *S* and the second
+ request for server *S* resolved to *IP2* but came from the cache, then the
+ log entry for the second request will show *IP2*.
+
+.. _admin-logging-fields-plugin:
+
+Plugin Details
+~~~~~~~~~~~~~~
+
+.. _piid:
+.. _pitag:
+
+Logging fields which may be used to obtain details of plugins involved in the
+transaction.
+
+===== ============ ============================================================
+Field Source Description
+===== ============ ============================================================
+piid Proxy Plugin Plugin ID for the current transaction. This is set for
+ plugin driven transactions via
+ :c:func:`TSHttpConnectWithPluginId`.
+pitag Proxy Plugin Plugin tag for the current transaction. This is set for
+ plugin driven transactions via
+ :c:func:`TSHttpConnectWithPluginId`.
+===== ============ ============================================================
+
+.. _admin-logging-fields-proto:
+
+Protocols and Versions
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. _cqhv:
+.. _cqpv:
+.. _csshv:
+.. _sshv:
+
+These logging fields may be used to determine which protocols and/or versions
+were in effect for a given event.
+
+===== ===================== ===================================================
+Field Source Description
+===== ===================== ===================================================
+cqhv Client Request Client request HTTP version.
+cqpv Client Request Client request protocol and version.
+csshv Cached Proxy Response Origin server's HTTP version from cached version of
+ the document in |TS| proxy cache.
+sshv Origin Response Origin server's response HTTP version.
+===== ===================== ===================================================
+
+.. _admin-logging-fields-request:
+
+Request Details
+~~~~~~~~~~~~~~~
+
+.. _cqtx:
+
+The following logging fields are used to obtain the actual HTTP request
+details.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cqtx Client Request The full HTTP client request text, minus headers, e.g.
+ ``GET http://www.company.com HTTP/1.0``. In reverse proxy
+ mode, |TS| logs rewritten/mapped URL (according to the
+ rules in :file:`remap.config`), not the pristine/unmapped
+ URL.
+===== ============== ==========================================================
+
+.. _admin-logging-fields-ssl:
+
+SSL / Encryption
+~~~~~~~~~~~~~~~~
+
+.. _cqssl:
+.. _cqssr:
+.. _cqssv:
+.. _cqssc:
+.. _pqssl:
+
+Fields which expose the use, or lack thereof, of specific SSL and encryption
+features.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cqssl Client Request SSL client request status indicates if this client
+ connection is over SSL.
+cqssr Client Request SSL session ticket reused status; indicates if the current
+ request hit the SSL session ticket and avoided a full SSL
+ handshake.
+cqssv Client Request SSL version used to communicate with the client.
+cqssc Client Request SSL Cipher used by |TS| to communicate with the client.
+pqssl Proxy Request Indicates whether the connection from |TS| to the origin
+ was over SSL or not.
+===== ============== ==========================================================
+
+.. _admin-logging-fields-status:
+
+Status Codes
+~~~~~~~~~~~~
+
+.. _cfsc:
+.. _csssc:
+.. _pfsc:
+.. _pssc:
+.. _sssc:
+
+These log fields provide a variety of status codes, some numeric and some as
+strings, relating to client, proxy, and origin transactions.
+
+===== ===================== ===================================================
+Field Source Description
+===== ===================== ===================================================
+cfsc Client Request Finish status code specifying whether the client
+ request to |TS| was successfully completed
+ (``FIN``) or interrupted (``INTR``).
+csssc Cached Proxy Response HTTP response status code of the origin server
+ response, as cached by |TS|.
+pfsc Proxy Request Finish status code specifying whether the proxy
+ request from |TS| to the origin server was
+ successfully completed (``FIN``), interrupted
+ (``INTR``), or timed out (``TIMEOUT``).
+pssc Proxy Response HTTP response status code sent by |TS| proxy to the
+ client.
+sssc Origin Response HTTP response status code sent by the origin server
+ to the |TS| proxy.
+===== ===================== ===================================================
+
+.. _admin-logging-fields-tcp:
+
+TCP Details
+~~~~~~~~~~~
+
+.. _cqtr:
+
+The following logging fields reveal information about the TCP layer of client,
+proxy, and origin server connections.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cqtr Client Request TCP reused status of the connection between the client and
+ |TS| proxy, indicating whether the request was delivered
+ through an already established connection.
+===== ============== ==========================================================
+
+.. _admin-logging-fields-time:
+
+Timestamps and Durations
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _cqtd:
+.. _cqtn:
+.. _cqtq:
+.. _cqts:
+.. _cqtt:
+.. _crat:
+.. _ms:
+.. _msdms:
+.. _stms:
+.. _stmsh:
+.. _stmsf:
+.. _sts:
+.. _ttms:
+.. _ttmsh:
+.. _ttmsf:
+.. _tts:
+
+The logging fields expose a variety of timing related information about client,
+proxy, and origin transactions. Variants of some of the fields provide timing
+resolution of the same underlying detail in milliseconds and seconds (both
+fractional and rounded-down integers). These variants are particularly useful
+in accomodating the emulation of other HTTP proxy softwares' logging formats.
+
+Other fields in this category provide variously formatted timestamps of
+particular events within the current transaction (e.g. the time at which a
+client request was received by |TS|).
+
+===== ======================= =================================================
+Field Source Description
+===== ======================= =================================================
+cqtd Client Request Client request timestamp. Specifies the date of
+ the client request in the format ``YYYY-MM-DD``
+ (four digit year, two digit month, two digit day
+ - with leading zeroes as necessary for the latter
+ two).
+cqtn Client Request Client request timestamp in the Netscape
+ timestamp format.
+cqtq Client Request The time at which the client request was received
+ expressed as fractional (floating point) seconds
+ since midnight January 1, 1970 UTC (epoch), with
+ millisecond resolution.
+cqts Client Request Same as cqtq_, but as an integer without
+ sub-second resolution.
+cqtt Client Request Client request timestamp in the 24-hour format
+ ``hh:mm:ss`` (two digit hour, minutes, and
+ seconds - with leading zeroes as necessary).
+crat Origin Response Retry-After time in seconds if specified in the
+ origin server response.
+ms Proxy Timestamp in milliseconds of a specific milestone
+ for this request. See note below about specifying
+ which milestone to use.
+msdms Proxy Difference in milliseconds between the timestamps
+ of two milestones. See note below about
+ specifying which miletones to use.
+stms Proxy-Origin Connection Time (in milliseconds) spent accessing the origin
+ server. Measured from the time the connection
+ between proxy and origin is established to the
+ time it was closed.
+stmsh Proxy-Origin Connection Same as stms_, but represented in hexadecimal.
+stmsf Proxy-Origin Connection Same as stms_, but in fractional (floating point)
+ seconds.
+sts Proxy-Origin Connection Same as stms_, but in integer seconds (no
+ sub-second precision).
+ttms Client-Proxy Connection Time in milliseconds spent by |TS| processing the
+ entire client request. Measured from the time the
+ connection between the client and |TS| proxy was
+ established until the last byte of the proxy
+ response was delivered to the client.
+ttmsh Client-Proxy Connection Same as ttms_, but represented in hexadecimal.
+ttmsf Client-Proxy Connection Same as ttms_, but in fraction (floating point)
+ seconds.
+tts Client Request Same as ttms_, but in integer seconds (no
+ sub-second precision).
+===== ======================= =================================================
+
+.. note::
+
+ Logging fields for transaction milestones require specifying which of the
+ milestones to use. Similar to how header logging fields are used, these log
+ fields take the milestone name(s) in between curly braces, immediately
+ before the logging field name, as so::
+
+ %<{Milestone field name}ms>
+ %<{Milestone field name1-Milestone field name2}msdms>
+
+ For more information on transaction milestones in |TS|, refer to the
+ documentation on :c:func:`TSHttpTxnMilestoneGet`.
+
+.. _admin-logging-fields-urls:
+
+URLs, Schemes, and Paths
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _cqu:
+.. _cquc:
+.. _cqup:
+.. _cqus:
+.. _cquuc:
+.. _cquup:
+.. _cquuh:
+
+These log fields allow capture of URLs, or components (such as schemes and
+paths), from transactions processed by |TS|.
+
+===== ============== ==========================================================
+Field Source Description
+===== ============== ==========================================================
+cqu Proxy Request URI of the client request to |TS| (a subset of cqtx_). In
+ reverse proxy mode, |TS| logs the rewritten/mapped URL
+ (according to the rules in :file:`remap.config`), not the
+ pristine/unmapped URL.
+cquc Client Request Canonical URL from the client request to |TS|. This field
+ differs from cqu_ by having its contents URL-escaped
+ (spaces and various other characters are replaced by
+ percent-escaped entity codes).
+cqup Proxy Request Path component from the remapped client request.
+cqus Client Request URL scheme from the client request.
+cquuc Client Request Canonical (prior to remapping) URL from client request.
+cquup Client Request Canonical (prior to remapping) path component from the
+ client request. Compare with cqup_.
+cquuh Client Request Unmapped URL host from the client request.
+===== ============== ==========================================================
+
+Log Field Slicing
+=================
+
+It is sometimes desirable to slice a log field to limit the length of a given
+log field's output.
+
+Log Field slicing can be specified as below::
+
+ %<field[start:end]>
+ %<{field}container[start:end]>
+
+Omitting the slice notation defaults to the entire log field.
+
+Slice notation only applies to a log field that is of type string and can not
+be applied to IPs or timestamp which are converted to string from integer.
+
+The below slice specifiers are allowed.
+
+``[start:end]``
+ Log field value from start through end-1
+``[start:]``
+ Log field value from start through the rest of the string
+``[:end]``
+ Log field value from the beginning through end-1
+``[:]``
+ Default - entire Log field
+
+Some examples below ::
+
+ '%<cqup>' // the whole characters of <cqup>.
+ '%<cqup>[:]' // the whole characters of <cqup>.
+ '%<cqup[0:30]>' // the first 30 characters of <cqup>.
+ '%<cqup[-10:]>' // the last 10 characters of <cqup>.
+ '%<cqup[:-5]>' // everything except the last 5 characters of <cqup>.
+
diff --git a/doc/admin-guide/monitoring/logging/index.en.rst b/doc/admin-guide/logging/index.en.rst
similarity index 64%
rename from doc/admin-guide/monitoring/logging/index.en.rst
rename to doc/admin-guide/logging/index.en.rst
index fe581b0..0cab174 100644
--- a/doc/admin-guide/monitoring/logging/index.en.rst
+++ b/doc/admin-guide/logging/index.en.rst
@@ -15,26 +15,27 @@
specific language governing permissions and limitations
under the License.
-.. include:: ../../../common.defs
+.. include:: ../../common.defs
-.. _admin-monitoring-logging:
+.. _admin-logging:
Logging
*******
-|TS| generates log files that contain information about every
-request it receives and every error it detects. This chapter will examine the
-various log features, the configuration formats and also examine the various
-pre-defined log formats that are available.
+|TS| generates log files that contain information about every request it
+receives and every error it detects. This chapter will examine the various
+logging features: from how to configure logging destinations, defining custom
+logging formats, establishing logfile rotation, and more. Common configuration
+scenarios are also provided.
.. toctree::
:maxdepth: 2
understanding.en
- managing-logs.en
- log-formats.en
- summary-logs.en
- log-collation.en
- pipes.en
- log-builder.en
+ destinations.en
+ formatting.en
+ cache-results.en
+ rotation.en
+ filters.en
+ examples.en
diff --git a/doc/admin-guide/monitoring/logging/managing-logs.en.rst b/doc/admin-guide/logging/rotation.en.rst
similarity index 67%
rename from doc/admin-guide/monitoring/logging/managing-logs.en.rst
rename to doc/admin-guide/logging/rotation.en.rst
index b5d8ecc..bdf4ccb 100644
--- a/doc/admin-guide/monitoring/logging/managing-logs.en.rst
+++ b/doc/admin-guide/logging/rotation.en.rst
@@ -15,84 +15,28 @@
specific language governing permissions and limitations
under the License.
-.. include:: ../../../common.defs
+.. include:: ../../common.defs
-.. _admin-monitoring-logging-managing:
+.. _admin-logging-rotation-retention:
-Managing Logs
-*************
+Log Rotation and Retention
+**************************
-|TS| enables you to control where event log files are located and how much
-space they can consume. Additionally, you can specify how to handle low disk
-space in the logging directory.
+Logging is a nearly indispensable part of any networked service, but especially
+with high traffic installations care needs to be taken to ensure that log files
+don't exhaust storage space and cause maintenance or outage nightmares.
-Choosing the Logging Directory
-==============================
+|TS| provides a two-pronged solution: log rotation (also called log rolling) to
+keep individual logs as manageable in size as possible for easier ingestion and
+analysis by humans and other programs, and log retention to keep logs from
+using more space than available and necessary.
-By default, Traffic Server writes all event log files in the ``logs``
-directory located in the directory where you installed |TS|. To change this
-location, adjust the value of :ts:cv:`proxy.config.log.logfile_dir` in
-:file:`records.config`. You will need to either restart |TS| or run the
-command :option:`traffic_ctl config reload` for changes to take effect.
+This section covers both features.
-Controlling Logging Space
-=========================
+.. _admin-logging-rotation:
-|TS| enables you to control the amount of disk space that the logging directory
-can consume. This allows the system to operate smoothly within a specified
-space window for a long period of time. After you establish a space limit,
-|TS| continues to monitor the space in the logging directory. When the free
-space dwindles to the headroom limit, it enters a low space state and takes the
-following actions:
-
-- If the autodelete option (discussed in `Rolling Logs`_) is enabled, then
- |TS| identifies previously-rolled log files (log files with the ``.old``
- extension). It starts deleting files one by one, beginning with the oldest
- file, until it emerges from the low state. |TS| logs a record of all deleted
- files in the system error log.
-
-- If the autodelete option is disabled or there are not enough old log files
- to delete for the system to emerge from its low space state, then |TS|
- issues a warning and continues logging until space is exhausted. When
- available space is consumed, event logging stops. |TS| resumes event logging
- when enough space becomes available for it to exit the low space state. To
- make space available, either explicitly increase the logging space limit or
- remove files from the logging directory manually.
-
-You can run a :manpage:`cron(8)` script in conjunction with |TS| to
-automatically remove old log files from the logging directory before |TS|
-enters the low space state. Relocate the old log files to a temporary
-partition, where you can run a variety of log analysis scripts. Following
-analysis, either compress the logs and move to an archive location, or simply
-delete them.
-
-|TS| periodically checks the amount of log space used against both
-the log space allocation configured by
-:ts:cv:`proxy.config.log.max_space_mb_for_logs` and the actual
-amount of space available on the disk partition. The used log space
-is calculated by summing the size of all files present in the logging
-directory and is published in the
-:ts:stat:`proxy.process.log.log_files_space_used` metric. The
-:ts:cv:`proxy.config.log.max_space_mb_headroom` configuration
-variable specifies an amount of headroom that is subtracted from
-the log space allocation. This can be tuned to reduce the risk of
-completely filling the disk partition.
-
-Setting Log File Management Options
------------------------------------
-
-To set log management options, follow the steps below:
-
-#. In the :file:`records.config` file, edit the following variables
-
- - :ts:cv:`proxy.config.log.max_space_mb_for_logs`
- - :ts:cv:`proxy.config.log.max_space_mb_headroom`
-
-#. Run the command :option:`traffic_ctl config reload` to apply the configuration
- changes.
-
-Rolling Logs
-============
+Rotation Options
+================
|TS| provides automatic log file rolling. At specific intervals during the day
or when log files reach a certain size, |TS| closes its current set of log
@@ -104,7 +48,7 @@ every six hours, and adjusting from there.
Log file rolling offers the following benefits:
-- It defines an consistent interval over which log analysis can be performed.
+- It defines a consistent interval over which log analysis can be performed.
- It keeps any single log file from becoming too large and helps to
keep the logging system within the specified space limits.
@@ -121,9 +65,10 @@ to easily identify log files. When |TS| rolls a log file, it saves and closes
the old file before it starts a new file. |TS| renames the old file to include
the following information:
-- The format of the file (such as ``squid.log``).
+- The original log file's name (such as ``access.log``).
-- The hostname of the |TS| that generated the log file.
+- The hostname of the |TS| node that generated the log file (useful in |TS|
+ clusters and log collation configurations).
- Two timestamps separated by a hyphen (``-``). The first timestamp is
a *lower bound* for the timestamp of the first record in the log
@@ -155,11 +100,9 @@ Format Description Sample
``%S`` The second in two-digit format, from 00-59. 36
====== ============================================ ======
-.. XXX can %S ever be 60, on account of leap seconds, or does ATS have leap-second related issues that otherwise interfere?
-
The following is an example of a rolled log filename: ::
- squid.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old
+ access.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old
The logging system buffers log records before writing them to disk. When
a log file is rolled, the log buffer might be partially full. If it is,
@@ -170,13 +113,13 @@ timestamp will be a lower bound for the timestamp of the first entry.
For example, suppose logs are rolled every three hours, and the first
rolled log file is: ::
- squid.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old
+ access.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old
If the lower bound for the first entry in the log buffer at 3:00:00 is
2:59:47, then the next log file will have the following timestamp when
rolled: ::
- squid.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old
+ access.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old
The contents of a log file are always between the two timestamps. Log
files do not contain overlapping entries, even if successive timestamps
@@ -227,39 +170,68 @@ they reach a certain size, adjust the following settings in
#. Run the command :option:`traffic_ctl config reload` to apply the configuration
changes.
-You can fine-tune log file rolling settings for a custom log file in the
+You can fine-tune log file rolling settings for individual log files in the
``log.*`` specification in :file:`logging.config`. The custom log file uses the
rolling settings provided in the relevant ``log`` function call, which override
the default settings you specify in Traffic Manager or :file:`records.config`
described above.
-.. _admin-monitoring-logging-host-split:
+.. _admin-logging-retention:
-Separating Logs by Origin
-=========================
+Retention Options
+=================
-The default :file:`log_hosts.config` file is located in the |TS| ``config``
-directory. To record HTTP transactions for different origin servers in separate
-log files, you must specify the hostname of each origin server on a separate
-line in :file:`log_hosts.config`. For example, if you specify the keyword
-``sports``, then Traffic Server records all HTTP transactions from
-``sports.yahoo.com`` and ``www.foxsports.com`` in a log file called
-``squid-sports.log`` (if the Squid format is enabled).
+|TS| enables you to control the amount of disk space that the logging directory
+can consume. This allows the system to operate smoothly within a specified
+space window for a long period of time. After you establish a space limit,
+|TS| continues to monitor the space in the logging directory. When the free
+space dwindles to the headroom limit, it enters a low space state and takes the
+following actions:
-.. important::
+- If the autodelete option is enabled, then |TS| identifies previously-rolled
+ log files (log files with the ``.old`` extension). It starts deleting files
+ one by one, beginning with the oldest file, until it emerges from the low
+ state. |TS| logs a record of all deleted files in the system error log.
- If |TS| is clustered and you enable log file collation, then you
- should use the same :file:`log_hosts.config` file on every |TS| node in the
- cluster.
+- If the autodelete option is disabled or there are not enough old log files
+ to delete for the system to emerge from its low space state, then |TS|
+ issues a warning and continues logging until the allocated log space is
+ exhausted (which, if configured appropriately, will be well before your
+ actual filesystem space is fully consumed and causes additional problems).
+ At this point, event logging stops even though proxy traffic is still served
+ without client-visible interruption. |TS| resumes event logging when enough
+ space becomes available for it to exit the low space state. To make space
+ available, either explicitly increase the logging space limit or remove
+ files from the logging directory manually.
-To edit the log hosts list:
+You can run a :manpage:`cron(8)` script in conjunction with |TS| to
+automatically remove old log files from the logging directory before |TS|
+enters the low space state. Relocate the old log files to a temporary
+partition, where you can run a variety of log analysis scripts. Following
+analysis, either compress the logs and move to an archive location, or simply
+delete them.
+
+|TS| periodically checks the amount of log space used against both the log
+space allocation configured by :ts:cv:`proxy.config.log.max_space_mb_for_logs`
+and the actual amount of space available on the disk partition. The used log
+space is calculated by summing the size of all files present in the logging
+directory and is published in the
+:ts:stat:`proxy.process.log.log_files_space_used` metric. The
+:ts:cv:`proxy.config.log.max_space_mb_headroom` configuration variable
+specifies an amount of headroom that is subtracted from the log space
+allocation. This can be tuned to reduce the risk of completely filling the disk
+partition.
-#. Enter the hostname of each origin server on a separate line in
- :file:`log_hosts.config`. ::
+Setting Log File Management Options
+-----------------------------------
- webserver1
- webserver2
- webserver3
+To set log management options, follow the steps below:
-#. Run the command :option:`traffic_ctl config reload` to apply the changes.
+#. In the :file:`records.config` file, edit the following variables
+
+ - :ts:cv:`proxy.config.log.max_space_mb_for_logs`
+ - :ts:cv:`proxy.config.log.max_space_mb_headroom`
+
+#. Run the command :option:`traffic_ctl config reload` to apply the configuration
+ changes.
diff --git a/doc/admin-guide/logging/understanding.en.rst b/doc/admin-guide/logging/understanding.en.rst
new file mode 100644
index 0000000..2594632
--- /dev/null
+++ b/doc/admin-guide/logging/understanding.en.rst
@@ -0,0 +1,161 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../../common.defs
+
+.. _admin-logging-understanding:
+
+Understanding |TS| Logs
+***********************
+
+|TS| records information about every transaction (or request) it processes and
+every error it detects in log files. This information is separated into various
+logs, which are discussed below.
+
+By analyzing the log files, you can determine how many people use the |TS|
+cache, how much information each person requested, what pages are most popular,
+and so on. You can analyze the standard format log files with off-the-shelf
+analysis packages. To help with log file analysis, you can separate log files
+so they contain information specific to protocol or hosts. You can also
+configure |TS| to roll log files automatically at specific intervals during the
+day or when they reach a certain size.
+
+Enabling Logs
+=============
+
+By default, |TS| creates both error and event log files and
+records system information in system log files. You can disable event
+logging and/or error logging by setting the configuration variable
+:ts:cv:`proxy.config.log.logging_enabled` in :file:`records.config`
+to one of the following values:
+
+===== =========================================================================
+Value Description
+===== =========================================================================
+``0`` Disable both event and error logging.
+``1`` Enable error logging only.
+``2`` Enable event logging only.
+``3`` Enable both event and error logging.
+===== =========================================================================
+
+.. _admin logging-types:
+
+Log Types
+=========
+
+Three separate classes of log files exist: `System Logs`_, `Error Logs`_, and
+`Event Logs`_. The fourth log type covered here, `Summary Logs`_ are a special
+instance of the event logs, but instead of including details of individual
+transactions, the summary logs allow you to emit log entries which aggregate
+all events that occur over arbitrary periods of time (the specific period of
+time being a fixed configuration of each summary log you create).
+
+.. _admin-logging-type-system:
+
+System Logs
+-----------
+
+System log files record system information, including messages about the state
+of |TS| and any errors or warnings it produces. This kind of information might
+include a note that event log files were rolled, a warning that cluster
+communication timed out, or an error indicating that |TS| was restarted. If
+|TS| is failing to start properly on your system(s), this is the first place
+you'll want to look for possible hints as to the cause.
+
+All system information messages are logged with the system-wide logging
+facility :manpage:`syslog` under the daemon facility. The
+:manpage:`syslog.conf(5)` configuration file (stored in the ``/etc`` directory)
+specifies where these messages are logged. A typical location is
+``/var/log/messages`` (Linux).
+
+The :manpage:`syslog(8)` process works on a system-wide basis, so it serves as
+the single repository for messages from all |TS| processes (including
+:program:`traffic_server`, :program:`traffic_manager`, and
+:program:`traffic_cop`).
+
+System information logs observe a static format. Each log entry in the log
+contains information about the date and time the error was logged, the hostname
+of the |TS| that reported the error, and a description of the error or warning.
+
+.. _admin-logging-type-error:
+
+Error Logs
+----------
+
+Error log files record information about why a particular transaction was in
+error. Refer to :ref:`admin-monitoring-errors` for a list of the messages
+logged by |TS|.
+
+.. _admin-logging-type-event:
+
+Event Logs
+----------
+
+Event log files (also called access logs) record information about the state of
+each transaction |TS| processes and form the true bulk of logging output in
+|TS| installations. Most of the remaining documentation in this chapter applies
+to creating, formatting, rotating, and filtering event logs.
+
+Individual event log outputs are configured in :file:`logging.config` and as
+such, the documentation provided in that configuration file's section should be
+consulted in concert with the sections of this chapter.
+
+.. _admin-logging-type-summary:
+
+Summary Logs
+------------
+
+Summary logs are an extension of the event logs, but instead of providing
+details for individual events, aggregate statistics are presented for all
+events occurring within the specified time window. Summary logs have access to
+all of the same fields as event logs, with the restriction that every field
+must be used within an aggregating function. Summary logs may not mix both
+aggregated and unaggregated fields.
+
+The aggregating functions available are:
+
+=========== ===================================================================
+Function Description
+=========== ===================================================================
+``AVERAGE`` Average (mean) of the given field's value from all events within
+ the interval. May only be used on numeric fields.
+``COUNT`` The total count of events which occurred within the interval. No
+ field name is necessary (``COUNT(*)`` may be used instead).
+``FIRST`` The value of the first event, chronologically, which was observed
+ within the interval. May be used with any type of field; numeric or
+ otherwise.
+``LAST`` The value of the last event, chronologically, which was observed
+ within the interval. May be used with any type of field; nuemric or
+ otherwise.
+``SUM`` Sum of the given field's value from all events within the interval.
+ May only be used on numeric fields.
+=========== ===================================================================
+
+Summary logs are defined in :file:`logging.config` just like regular event
+logs, with the only two differences being the exclusive use of the
+aforementioned aggregate functions and the specification of an interval, as so:
+
+.. code:: lua
+
+ mysummary = format {
+ Format = "%<operator(field)> : %<operator(field)>",
+ Interval = "n"
+ }
+
+The interval itself is given with *n* as the number of seconds for each period
+of aggregation. There is no default value.
+
diff --git a/doc/admin-guide/monitoring/error-messages.en.rst b/doc/admin-guide/monitoring/error-messages.en.rst
index 1f36c71..61094d4 100644
--- a/doc/admin-guide/monitoring/error-messages.en.rst
+++ b/doc/admin-guide/monitoring/error-messages.en.rst
@@ -62,7 +62,7 @@ Process Warnings
``Log format symbol <symbol name> not found``
Custom log format references a field symbol that does not exist.
- Refer to :ref:`admin-monitoring-logging-formats`.
+ Refer to :ref:`admin-logging-formats`.
``Missing field for field marker``
Error reading a log buffer.
@@ -125,7 +125,7 @@ Alarm Messages
``Traffic Server failed to parse line <line number> of the logging config file <filename>``
Check your custom log configuration file; there could be syntax
- errors. Refer to :ref:`custom-logging-fields` for correct custom log format fields.
+ errors. Refer to :ref:`admin-logging-fields` for correct custom log format fields.
``vip_config binary is not setuid root, manager will be unable to enable virtual ip addresses``
The :program:`traffic_manager` process is not able to set virtual IP
@@ -158,7 +158,7 @@ look for a file named ``apache_cache#read_error`` in the template table. This ca
out error message sets or, because this variable is overridable, to select an error message set
based on data in the transaction.
-The text for an error message is processed as if it were a :ref:`custom logging format <custom-logging-fields>` which
+The text for an error message is processed as if it were a :ref:`admin-logging-fields` which
enables customization by values present in the transaction for which the error occurred.
The following table lists the hard-coded Traffic Server HTTP messages,
diff --git a/doc/admin-guide/monitoring/index.en.rst b/doc/admin-guide/monitoring/index.en.rst
index adf3212..bd51dba 100644
--- a/doc/admin-guide/monitoring/index.en.rst
+++ b/doc/admin-guide/monitoring/index.en.rst
@@ -19,13 +19,12 @@
.. _admin-monitoring:
-Event and Error Monitoring
-**************************
+Monitoring
+**********
.. toctree::
:maxdepth: 3
- logging/index.en
alarms.en
statistics/index.en
monitoring/index.en
diff --git a/doc/admin-guide/monitoring/logging/log-collation.en.rst b/doc/admin-guide/monitoring/logging/log-collation.en.rst
deleted file mode 100644
index 3334e7f..0000000
--- a/doc/admin-guide/monitoring/logging/log-collation.en.rst
+++ /dev/null
@@ -1,142 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-.. include:: ../../../common.defs
-
-.. _admin-monitoring-logging-log-collation:
-
-Log Collation
-*************
-
-You can use the |TS| log file collation feature to collect all logged
-information in one place. Log collation enables you to analyze a set of |TS|
-clustered nodes as a whole (rather than as individual nodes) and to use a large
-disk that might only be located on one of the nodes in the cluster.
-
-|TS| collates log files by using one or more nodes as log collation servers and
-all remaining nodes as log collation clients. When a |TS| node generates a
-buffer of event log entries, it first determines if it is the collation server
-or a collation client. The collation server node writes all log buffers to its
-local disk, just as it would if log collation was not enabled. Log collation
-servers can be part of a node running |TS|.
-
-The collation client nodes prepare their log buffers for transfer across the
-network and send the buffers to the log collation server. When the log
-collation server receives a log buffer from a client, it writes it to its own
-log file as if it was generated locally. For a visual representation of this,
-see the figure below.
-
-.. figure:: /static/images/admin/logcolat.jpg
- :align: center
- :alt: Log collation
-
- Log collation
-
-If log clients cannot contact their log collation server, then they write their
-log buffers to their local disks, into *orphan* log files. Orphaned log files
-require manual collation.
-
-.. note::
-
- Log collation can have an impact on network performance. Because all nodes
- are forwarding their log data buffers to the single collation server, a
- bottleneck can occur. In addition, collated log files contain timestamp
- information for each entry, but entries in the files do not appear in
- strict chronological order. You may want to sort collated log files before
- doing analysis.
-
-.. _admin-configuring-traffic-server-to-be-a-collation-server:
-
-Server Configuration
-====================
-
-To configure a |TS| node to be a collation server, perform the following
-configuration adjustments in :file:`records.config`:
-
-#. Set :ts:cv:`proxy.local.log.collation_mode` to ``1`` to indicate this node
- will be a server. ::
-
- CONFIG proxy.local.log.collation_mode INT 1
-
-#. Configure the port on which the server will listen to incoming collation
- transfers from clients, using :ts:cv:`proxy.config.log.collation_port`. If
- omitted, this defaults to port ``8085``. ::
-
- CONFIG proxy.config.log.collation_port INT 8085
-
-#. Configure the shared secret used by collation clients to authenticate their
- sessions, using :ts:cv:`proxy.config.log.collation_secret`. ::
-
- CONFIG proxy.config.log.collation_secret STRING "seekrit"
-
-#. Run the command :option:`traffic_ctl config reload` to apply the configuration
- changes.
-
-.. note::
-
- If you modify the ``collation_port`` or ``secret`` after connections
- between the collation server and collation clients have been established,
- then you must restart Traffic Server on all nodes.
-
-.. _admin-monitoring-logging-collation-client:
-
-Client Configuration
-====================
-
-To configure a |TS| node to be a collation client, follow the steps below. If
-you modify the ``collation_port`` or ``secret`` after connections between the
-collation clients and the collation server have been established, then you must
-restart |TS|.
-
-#. In the :file:`records.config` file, edit the following variables:
-
- - :ts:cv:`proxy.local.log.collation_mode`: ``2`` to configure this node as
- log collation client and send standard formatted log entries to the
- collation server. For custom log entries, see :file:`logging.config`.
- - :ts:cv:`proxy.config.log.collation_host`
- - :ts:cv:`proxy.config.log.collation_port`
- - :ts:cv:`proxy.config.log.collation_secret`
- - :ts:cv:`proxy.config.log.collation_host_tagged`
- - :ts:cv:`proxy.config.log.max_space_mb_for_orphan_logs`
-
-#. Run the command :option:`traffic_ctl config reload` to apply the configuration
- changes.
-
-Collating Custom Logs
-=====================
-
-If you use custom event log files, then you must edit :file:`logging.config`,
-in addition to configuring a collation server and collation clients.
-
-To collate custom event log files:
-
-#. On each collation client, edit :file:`logging.config` and add the
- ``CollationHosts`` attribute to the relevant logs. For example, adding two
- collation hosts to an ASCII log that uses the Squid format would look like:
-
- .. code:: lua
-
- log.ascii {
- Format = squid,
- Filename = 'squid',
- CollationHosts = { '192.168.1.100:4567', '192.168.1.101:4567' }
- }
-
-#. Run the command :option:`traffic_ctl config reload` to restart Traffic Server on the
- local node or :option:`traffic_ctl config reload` to restart Traffic Server on all
- the nodes in a cluster.
-
diff --git a/doc/admin-guide/monitoring/logging/log-formats.en.rst b/doc/admin-guide/monitoring/logging/log-formats.en.rst
deleted file mode 100644
index b1bfbc9..0000000
--- a/doc/admin-guide/monitoring/logging/log-formats.en.rst
+++ /dev/null
@@ -1,1128 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-.. include:: ../../../common.defs
-
-.. _admin-monitoring-logging-formats:
-
-Log Formats
-***********
-
-This document provides a reference for all the different logging formats |TS|
-supports. Rather than just reading about those formats, you may also want to
-try our `online event log builder <http://trafficserver.apache.org/logbuilder/>`_
-for an interactive way of building and understanding log formats.
-
-.. _admin-logging-binary-v-ascii:
-
-Binary or ASCII?
-================
-
-You can configure |TS| to create event log files in either of the following:
-
-ASCII
- These files are human-readable and can be processed using standard,
- off-the-shelf log analysis tools. However, |TS| must perform additional
- processing to create the files in ASCII, which mildly impacts system
- overhead. ASCII files also tend to be larger than the equivalent binary
- files. By default, ASCII log files have a ``.log`` filename extension.
-
-Binary
- These files generate lower system overhead and generally occupy less space
- on the disk than ASCII files (depending on the type of information being
- logged). However, you must use a converter application before you can read
- or analyze binary files via standard tools. By default, binary log files use
- a ``.blog`` filename extension.
-
-While binary log files typically require less disk space, there are exceptions.
-
-For example, the value ``0`` (zero) requires only one byte to store in ASCII,
-but requires four bytes when stored as a binary integer. Conversely, if you
-define a custom format that logs IP addresses, then a binary log file would
-only require four bytes of storage per 32-bit address. However, the same IP
-address stored in dot notation would require around 15 characters (bytes) in an
-ASCII log file.
-
-It is wise to consider the type of data that will be logged before you select
-ASCII or binary for your log files, if your decision is being driven by storage
-space concerns. For example, you might try logging to both formats
-simultaneously for a representative period of time and compare the storage
-requirements of the two logs to determine whether one or the other provides
-any measurable savings.
-
-Defining Log Objects
-====================
-
-To perform any logging at all on your |TS| nodes, you must have at least one
-log defined in :file:`logging.config`. These definitions configure
-what logs will be created, the format they will use (covered in the sections
-:ref:`admin-monitoring-logging-format-standard` and
-:ref:`admin-monitoring-logging-format-custom`), any filters which may be
-applied to events before logging them, and when or how the logs will be rolled.
-
-Log Filters
-===========
-
-Filters, configured in :file:`logging.config` allow you to create filters,
-which may be applied to log definitions, limiting the types of entries which
-will be included in the log output. This may be useful if your |TS| nodes
-receive many events which you have no need to log or analyze.
-
-.. _admin-monitoring-logging-format-standard:
-
-Standard Formats
-================
-
-The standard log formats include Squid, Netscape Common, Netscape extended, and
-Netscape Extended-2. The standard log file formats can be analyzed with a wide
-variety of off-the-shelf log-analysis packages. You should use one of the
-standard event log formats unless you need information that these formats do
-not provide.
-
-These formats may be used by editing :file:`logging.config`.
-
-.. _admin-logging-format-squid:
-
-Squid
------
-
-The following figure shows a sample log entry in a Squid log file.
-
-.. figure:: /static/images/admin/squid_format.jpg
- :align: center
- :alt: Sample log entry in squid.log
-
- Sample log entry in squid.log
-
-====== ========= ==============================================================
-Field Symbol Description
-====== ========= ==============================================================
-1 cqtq The client request timestamp in Squid format. The time of the
- client request in seconds since January 1, 1970 UTC (with
- millisecond resolution).
-2 ttms The time |TS| spent processing the client request. The number
- of milliseconds between the time the client established the
- connection with |TS| and the time |TS| sent the last byte of
- the response back to the client.
-3 chi The IP address of the client\u2019s host machine.
-4 crc/pssc The cache result code; how the cache responded to the request:
- ``HIT``, ``MISS``, and so on. Cache result codes are described
- in :ref:`admin-monitoring-logging-cache-result-codes`. The
- proxy response status code (HTTP response status code from
- |TS| to client).
-5 psql The length of the |TS| response to the client in bytes,
- including headers and content.
-6 cqhm The client request method: ``GET``, ``POST``, and so on.
-7 cauc The client request canonical URL; blanks and other characters
- that might not be parsed by log analysis tools are replaced by
- escape sequences. The escape sequence is a percentage sign
- followed by the ASCII code number of the replaced character in
- hex.
-8 caun The username of the authenticated client. A hyphen (``-``)
- means that no authentication was required.
-9 phr/pqsn The proxy hierarchy route. The route |TS| used to retrieve the
- object.
-10 psct The proxy response content type. The object content type taken
- from the |TS| response header.
-====== ========= ==============================================================
-
-.. _admin-logging-format-common:
-
-Netscape Common
----------------
-
-The following figure shows a sample log entry in a Netscape Common log file.
-
-.. figure:: /static/images/admin/netscape_common_format.jpg
- :align: center
- :alt: Sample log entry in common.log
-
- Sample log entry in common.log
-
-====== ========= ==============================================================
-Field Symbol Description
-====== ========= ==============================================================
-1 chi The IP address of the client's host machine.
-2 -- This hyphen (``-``) is always present in Netscape log entries.
-3 caun The authenticated client username. A hyphen (``-``) means no
- authentication was required.
-4 cqtd The date and time of the client request, enclosed in brackets.
-5 cqtx The request line, enclosed in quotes.
-6 pssc The proxy response status code (HTTP reply code).
-7 pscl The length of the |TS| response to the client in bytes.
-====== ========= ==============================================================
-
-.. _admin-logging-format-extended:
-
-Netscape Extended
------------------
-
-The following figure shows a sample log entry in a Netscape Extended log file.
-
-.. figure:: /static/images/admin/netscape_extended_format.jpg
- :align: center
- :alt: Sample log entry in extended.log
-
- Sample log entry in extended.log
-
-In addition to field 1-7 from the Netscape Common log format above, the
-Extended format also adds the following fields:
-
-====== ========= ==============================================================
-Field Symbol Description
-====== ========= ==============================================================
-8 sssc The origin server response status code.
-9 sshl The server response transfer length; the body length in the
- origin server response to |TS|, in bytes.
-10 cqcl The client request transfer length; the body length in the
- client request to |TS|, in bytes.
-11 pqcl The proxy request transfer length; the body length in the |TS|
- request to the origin server.
-12 cqhl The client request header length; the header length in the
- client request to |TS|.
-13 pshl The proxy response header length; the header length in the
- |TS| response to the client.
-14 pqhl The proxy request header length; the header length in |TS|
- request to the origin server.
-15 sshl The server response header length; the header length in the
- origin server response to |TS|.
-16 tts The time |TS| spent processing the client request; the number
- of seconds between the time that the client established the
- connection with |TS| and the time that |TS| sent the last byte
- of the response back to the client.
-====== ========= ==============================================================
-
-.. _admin-logging-format-extended2:
-
-Netscape Extended2
-------------------
-
-The following figure shows a sample log entry in a Netscape Extended2 log file.
-
-.. figure:: /static/images/admin/netscape_extended2_format.jpg
- :align: center
- :alt: Sample log entry in extended2.log
-
- Sample log entry in extended2.log
-
-In addition to field 1-16 from the log formats above, the Extended2 format also adds
-the following fields:
-
-====== ======== ===============================================================
-Field Symbol Description
-====== ======== ===============================================================
-17 phr The proxy hierarchy route; the route |TS| used to retrieve
- the object.
-18 cfsc The client finish status code: ``FIN`` if the client request
- completed successfully or ``INTR`` if the client request was
- interrupted.
-19 pfsc The proxy finish status code: ``FIN`` if the |TS| request to
- the origin server completed successfully or ``INTR`` if the
- request was interrupted.
-20 crc The cache result code; how the |TS| cache responded to the
- request: ``HIT``, ``MISS``, and so on. Cache result codes are
- listed in :ref:`admin-monitoring-logging-cache-result-codes`.
-====== ======== ===============================================================
-
-.. _admin-monitoring-logging-format-custom:
-
-Custom Formats
-==============
-
-Defining a Format
------------------
-
-Custom logging formats in |TS| are defined by editing :file:`logging.config`
-and adding new format entries for each format you wish to define. The syntax is
-fairly simple: every format must contain a ``Format`` attribute, which is the
-string defining the format of each line in the log, and may also contain an
-optional ``Interval`` attribute defining the log aggregation interval for
-any logs which use the format (see :ref:`admin-monitoring-logging-summary-logs`
-for more information).
-
-The return value from the ``format`` function is the log format object which
-may then be supplied to the appropriate ``log.*`` functions that define your
-logging destinations.
-
-A very simple exampe, which contains only the timestamp of when the event began
-and the canonical URL of the request, would look like:
-
-.. code:: lua
-
- myformat = format {
- Format = "%<cqtq> %<cauc>"
- }
-
-You may include as many custom field codes as you wish. The full list of codes
-available can be found in :ref:`custom-logging-fields`. You may also include
-any literal characters in your format. For example, if we wished to separate
-the timestamp and canonical URL in our customer format above with a slash
-instead of a space, or even a slash surrounded by spaces, we could do so by
-just adding the desired characters to the format string:
-
-.. code:: lua
-
- myformat = format {
- Format = "%<cqtq> / %<cauc>"
- }
-
-You may define as many custom formats as you wish. To apply changes to custom
-formats, you will need to run the command :option:`traffic_ctl config reload`
-after saving your changes to :file:`logging.config`.
-
-.. _custom-logging-fields:
-
-Custom Logging Fields
----------------------
-
-The following list describes |TS| custom logging fields.
-
-.. _cqh:
-
-``{HTTP header field name}cqh``
- Logs the information in the requested field of the client request
- HTTP header. For example, ``%<{Accept-Language}cqh>`` logs the
- ``Accept-Language:`` field in client request headers.
-
- .. note::
- ecqh is the URL-encoded version of this map
-
-.. _pqh:
-
-``{HTTP header field name}pqh``
- Logs the information in the requested field of the proxy request
- HTTP header. For example, ``%<{Authorization}pqh>`` logs
- the ``Authorization:`` field in proxy request headers.
-
- .. note::
- epqh is the URL-encoded version of this map
-
-.. _psh:
-
-``{HTTP header field name}psh``
- Logs the information in the requested field of the proxy response
- HTTP header. For example, ``%<{Retry-After}psh>`` logs the
- ``Retry-After:`` field in proxy response headers.
-
- .. note::
- epsh is the URL-encoded version of this map
-
-.. _ssh:
-
-``{HTTP header field name}ssh``
- Logs the information in the requested field of the server response
- HTTP header. For example, ``%<{Age}ssh>`` logs the ``Age:`` field in
- server response headers.
-
- .. note::
- essh is the URL-encoded version of this map
-
-.. _cssh:
-
-``{HTTP header field name}cssh``
- Logs the information in the requested field of the cached server response
- HTTP header. For example, ``%<{Age}cssh>`` logs the ``Age:`` field in
- the cached server response headers.
-
- .. note::
- ecssh is the URL-encoded version of this map
-
-.. _caun:
-
-``caun``
- The client authenticated username; result of the RFC931/ident lookup
- of the client username.
-
-.. _cfsc:
-
-``cfsc``
- The client finish status code; specifies whether the client request
- to |TS| was successfully completed (``FIN``) or
- interrupted (``INTR``).
-
-.. _chi:
-
-``chi``
- The IP address of the client's host machine.
-
-.. _chih:
-
-``chih``
- The IP address of the client's host machine in hexadecimal.
-
-.. _hii:
-
-``hii``
- This is the incoming (interface) IP address for |TS|, in
- otherwords this is the IP address the client connected to.
-
-.. _hiih:
-
-``hiih``
- The the incoming (interface) IP address in hexadecimal.
-
-.. _chp:
-
-``chp``
- The port number of the client's host machine.
-
-.. _cqcl:
-
-``cqcl``
- The client request content length; the body length in the client
- request to |TS| (in bytes).
-
-.. _cqhl:
-
-``cqhl``
- The client request header length; the header length in the client
- request to |TS|.
-
-.. _cqql:
-
-``cqql``
- The total client request length; the header length and content length of
- the client request to |TS|.
-
-.. _cqhm:
-
-``cqhm``
- The HTTP method in the client request to |TS|: ``GET``,
- ``POST``, and so on (subset of ``cqtx``).
-
-.. _cqhv:
-
-``cqhv``
- The client request HTTP version.
-
-.. _cqpv:
-
-``cqpv``
- The client request protocol and version.
-
-.. _cqtd:
-
-``cqtd``
- The client request timestamp. Specifies the date of the client
- request in the format yyyy-mm-dd, where yyyy is the 4-digit year, mm
- is the 2-digit month, and dd is the 2-digit day.
-
-.. _cqtn:
-
-``cqtn``
- The client request timestamp; date and time of the client's request
- (in the Netscape timestamp format).
-
-.. _cqtq:
-
-``cqtq``
- The time of the client request since January 1, 1970 UTC (epoch),
- with millisecond resolution.
-
-.. _cqts:
-
-``cqts``
- The time of the client request since January 1, 1970 UTC (EPOCH), with second resolution.
-
-.. _cqtt:
-
-``cqtt``
- The client request timestamp. The time of the client request in the
- format hh:mm:ss, where hh is the two-digit hour in 24-hour format,
- mm is the two-digit minutes value, and ss is the 2-digit seconds
- value (for example, 16:01:19).
-
-.. _cqtr:
-
-``cqtr``
- The TCP reused status; indicates if this client request went through
- an already established connection.
-
-.. _cqssl:
-
-``cqssl``
- The SSL client request status indicates if this client connection
- is over SSL.
-
-.. _cqssr:
-
-``cqssr``
- The SSL session ticket reused status; indicates if this request hit
- the SSL session ticket and avoided a full SSL handshake.
-
-.. _cqssv:
-
-``cqssv``
- The SSL version used to communicate with the client.
-
-.. _cqssc:
-
-``cqssc``
- The cipher used by |TS| to communicate with the client over SSL.
-
-.. _cqtx:
-
-``cqtx``
- The full HTTP client request text, minus headers; for example, ::
-
- GET http://www.company.com HTTP/1.0
-
- In reverse proxy mode, |TS| logs the rewritten/mapped URL
- (according to the rules in :file:`remap.config`), _not_ the
- pristine/unmapped URL.
-
-.. _cqu:
-
-``cqu``
- The universal resource identifier (URI) of the request from client
- to |TS| (subset of ``cqtx`` ).
-
- In reverse proxy mode, |TS| logs the rewritten/mapped URL
- (according to the rules in :file:`remap.config`), _not_ the
- pristine/unmapped URL.
-
-.. _cquc:
-
-``cquc``
- The client request canonical URL. This differs from ``cqu`` in that
- blanks (and other characters that might not be parsed by log
- analysis tools) are replaced by escape sequences. The escape
- sequence is a percentage sign followed by the ASCII code number in
- hex.
-
- See `cquuc`_.
-
-.. _cqup:
-
-``cqup``
- The client request URL path; specifies the argument portion of the
- URL (everything after the host). For example, if the URL is
- ``http://www.company.com/images/x.gif``, then this field displays
- ``/images/x.gif``
-
- See `cquup`_.
-
-.. _cqus:
-
-``cqus``
- The client request URL scheme.
-
-.. _cquuc:
-
-``cquuc``
- The client request unmapped URL canonical. This field records a URL
- before it is remapped (reverse proxy mode).
-
-.. _cquup:
-
-``cquup``
- The client request unmapped URL path. This field records a URL path
- before it is remapped (reverse proxy mode).
-
-.. _cquuh:
-
-``cquuh``
- The client request unmapped URL host. This field records a URL's
- host before it is remapped (reverse proxy mode).
-
-.. crid:
-
-``crid``
- This is the sequence number of this client request. This starts over at
- ``0`` on every server restart.
-
-.. cruuid:
-``cruuid``
- This is a UUID for the client request, uniquely identifying this
- transaction. This is actually a concatenation of the ``puuid`` and the
- ``crid``.
-
-.. _cluc:
-
-``cluc``
- The cache lookup URL, or cache key, for the client request. This URL is
- canonicalized as well.
-
-.. _crat:
-
-``crat``
- The Retry-After time in seconds, if specified by the origin server.
-
-.. _crc:
-
-``crc``
- The cache result code; specifies how the cache responded to the
- request (``HIT``, ``MISS``, and so on).
-
-.. _chm:
-
-``chm``
- The cache hit-miss status, specifying which level of the cache this
- was served out of. This is useful for example to show whether it was a
- RAM cache vs disk cache hit. Future versions of the cache will support
- more levels, but right now it only supports RAM (``HIT_RAM``) vs
- rotational disk (``HIT_DISK``).
-
-.. _csscl:
-
-``csscl``
- The cached body length (in bytes) from origin server to Traffic
- Server.
-
-.. _csshl:
-
-``csshl``
- The cached header length in the origin server response to |TS| (in bytes).
-
-.. _cssql:
-
-``cssql``
- The total cached response length; the header length and content
- length of a cached origin server response.
-
-.. _csshv:
-
-``csshv``
- The cached server response HTTP version (1.0, 1.1, etc.).
-
-.. _csssc:
-
-``csssc``
- The cached HTTP response status code from origin server to Traffic
- Server.
-
-.. _cwr:
-
-``cwr``
- The cache write result (``-``, ``WL_MISS``, ``INTR``, ``ERR`` or ``FIN``)
-
-.. _cwtr:
-
-``cwtr``
- The cache write transform result
-
-.. _fsiz:
-
-``fsiz``
- The size of the file (*n* bytes) as seen by the origin server.
-
-.. _ms:
-
-``{Milestone field name}ms``
- The timestamp in milliseconds of a specific milestone for this request.
- see :c:func:`TSHttpTxnMilestoneGet` for milestone names.
-
-.. _msdms:
-
-``{Milestone field name1-Milestone field name2}msdms``
- The difference in milliseconds of between two milestones.
- see :c:func:`TSHttpTxnMilestoneGet` for milestone names.
-
-.. _pfsc:
-
-``pfsc``
- The proxy finish status code; specifies whether the |TS|
- request to the origin server was successfully completed (``FIN``),
- interrupted (``INTR``) or timed out (``TIMEOUT``).
-
-.. puuid:
-
-``puuid``
- A UUID unique for the currently running :program:`traffic_server`
- process. This is generated on every server startup.
-
-.. _phn:
-
-``phn``
- The hostname of the |TS| that generated the log entry in
- collated log files.
-
-.. _phi:
-
-``phi``
- The IP of the |TS| that generated the log entry in
- collated log files.
-
-.. _phr:
-
-``phr``
- The proxy hierarchy route; the route |TS| used to retrieve
- the object.
-
-.. _php:
-
-``php``
- The TCP port number that |TS| served this request from.
-
-.. _piid:
-
-``piid``
- The plugin ID for the transaction. This is set for plugin driven transactions via :c:func:`TSHttpConnectWithPluginId`.
-
-.. _pitag:
-
-``pitag``
- The plugin tag for the transaction. This is set for plugin driven
- transactions via :c:func:`TSHttpConnectWithPluginId`.
-
-.. _pqcl:
-
-``pqcl``
- The proxy request content length; the body length in Traffic
- Server's request to the origin server.
-
-.. _pqhl:
-
-``pqhl``
- The proxy request header length; the header length in Traffic
- Server's request to the origin server.
-
-.. _pqql:
-
-``pqql``
- The total proxy request length; the header length and content length
- of Traffic Server's request to the origin server.
-
-.. _pqsi:
-
-``pqsi``
- The proxy request server IP address (0 on cache hits and parent-ip
- for requests to parent proxies).
-
-.. _pqsp:
-
-``pqsp``
- The proxy request server port (0 on cache hits and parent port
- for requests to parent proxies).
-
-.. _pqsn:
-
-``pqsn``
- The proxy request server name; the name of the server that fulfilled
- the request.
-
-.. _pscl:
-
-``pscl``
- The length of the |TS| response to the client (in bytes).
-
-.. _psct:
-
-``psct``
- The content type of the document from server response header: (for
- example, ``img/gif`` ).
-
-.. _pshl:
-
-``pshl``
- The header length in |TS|'s response to the client.
-
-.. _psql:
-
-``psql``
- The proxy response transfer length in Squid format (includes header
- and content length).
-
-.. _pssc:
-
-``pssc``
- The HTTP response status code from |TS| to the client.
-
-.. _pqssl:
-
-``pqssl``
- Indicates whether the connection from |TS| to the origin
- was over SSL or not.
-
-.. _sca:
-
-``sca``
- The number of attempts in the transaction Traffic Server tries to
- connect to the origin server.
-
-.. _shi:
-
-``shi``
- The IP address resolved from the DNS name lookup of the host in the
- request. For hosts with multiple IP addresses, this field records
- the IP address resolved from that particular DNS lookup.
-
- This can be misleading for cached documents. For example: if the
- first request was a cache miss and came from *IP1* for server
- *S* and the second request for server *S* resolved to
- *IP2* but came from the cache, then the log entry for the
- second request will show *IP2*.
-
-.. _shn:
-
-``shn``
- The hostname of the origin server.
-
-.. _sscl:
-
-``sscl``
- The body length (in bytes) from origin server to |TS|.
-
-.. _sshl:
-
-``sshl``
- The header length (in bytes) in the origin server response to |TS|.
-
-.. _ssql:
-
-``ssql``
- The total server response length; the header length and content length
- of the origin server response to |TS|.
-
-.. _sshv:
-
-``sshv``
- The server response HTTP version (1.0, 1.1, etc.).
-
-.. _sssc:
-
-``sssc``
- The HTTP response status code from origin server to |TS|.
-
-.. _stms:
-
-``stms``
- The time spent accessing the origin (in milliseconds); the time is
- measured from the time the connection with the origin is established
- to the time the connection is closed.
-
-.. _stmsh:
-
-``stmsh``
- Same as ``stms`` but in hexadecimal.
-
-.. _stmsf:
-
-``stmsf``
- The time |TS| spends accessing the origin as a fractional
- number of seconds. That is, the time is formated as a floating-point
- number, instead of an integer as in ``stms``.
-
- For example: if the time is 1500 milliseconds, then this field
- displays 1.5 while the ``stms`` field displays 1500 and the ``sts``
- field displays 1.
-
-.. _sts:
-
-``sts``
- The time |TS| spends accessing the origin, in seconds.
-
-.. _sstc:
-
-``sstc``
- The number of transactions between |TS| and the origin server
- from a single server session. A value greater than 0 indicates connection
- reuse.
-
-.. _ttms:
-
-``ttms``
- The time |TS| spends processing the client request; the
- number of milliseconds between the time the client establishes the
- connection with |TS| and the time |TS| sends the
- last byte of the response back to the client.
-
-.. _ttmsh:
-
-``ttmsh``
- Same as ``ttms`` but in hexadecimal.
-
-.. _ttmsf:
-
-``ttmsf``
- The time |TS| spends processing the client request as a
- fractional number of seconds. Time is specified in millisecond
- resolution; however, instead of formatting the output as an integer
- (as with ``ttms``), the display is formatted as a floating-point
- number representing a fractional number of seconds.
-
- For example: if the time is 1500 milliseconds, then this field
- displays 1.5 while the ``ttms`` field displays 1500 and the ``tts``
- field displays 1.
-
-.. _tts:
-
-``tts``
- The time |TS| spends processing the client request; the
- number of seconds between the time at which the client establishes
- the connection with |TS| and the time at which Traffic
- Server sends the last byte of the response back to the client.
-
-.. _logging-format-cross-reference:
-
-Custom Field Cross-Reference
-----------------------------
-
-The following sections illustrate the correspondence between |TS| custom
-logging fields and standard logging fields for the Squid and Netscape formats.
-
-Squid
-~~~~~
-
-The following is a list of the Squid logging fields and the
-corresponding logging field symbols.
-
-============== =============
-Squid Field Symbols
-============== =============
-time `cqts`_
-elapsed `ttms`_
-client `chi`_
-action/code `crc`_/`pssc`_
-size `psql`_
-method `cqhm`_
-url `cquc`_
-ident `caun`_
-hierarchy/from `phr`_/`pqsn`_
-content `psct`_
-============== =============
-
-This is the equivalent XML configuration for the log above::
-
- <LogFormat>
- <Name = "squid"/>
- <Format = "%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc>
- %<caun> %<phr>/%<pqsn> %<psct>"/>
- </LogFormat>
-
-.. _admin-log-formats-netscape-common:
-
-Netscape Common
-~~~~~~~~~~~~~~~
-
-The following is a list of the Netscape Common logging fields and the
-corresponding |TS| logging field symbols.
-
-=============== =============
-Netscape Common Field Symbols
-=============== =============
-host `chi`_
-usr `caun`_
-[time] [`cqtn`_]
-"req" "`cqtx`_"
-s1 `pssc`_
-c1 `pscl`_
-=============== =============
-
-This is the equivalent XML configuration for the log above::
-
- <LogFormat>
- <Name = "common"/>
- <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>"/>
- </LogFormat>
-
-.. _admin-log-formats-netscape-extended:
-
-Netscape Extended
-~~~~~~~~~~~~~~~~~
-
-The following table lists the Netscape Extended logging fields and the
-corresponding |TS| logging field symbols.
-
-================= =============
-Netscape Extended Field Symbols
-================= =============
-host `chi`_
-usr `caun`_
-[time] [`cqtn`_]
-"req" "`cqtx`_"
-s1 `pssc`_
-c1 `pscl`_
-s2 `sssc`_
-c2 `sscl`_
-b1 `cqcl`_
-b2 `pqcl`_
-h1 `cqhl`_
-h2 `pshl`_
-h3 `pqhl`_
-h4 `sshl`_
-xt `tts`_
-================= =============
-
-This is the equivalent XML configuration for the log above::
-
- <LogFormat>
- <Name = "extended"/>
- <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>
- %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts>"/>
- </LogFormat>
-
-.. _admin-log-formats-netscape-extended2:
-
-Netscape Extended-2
-~~~~~~~~~~~~~~~~~~~
-
-The following is a list of the Netscape Extended-2 logging fields and
-the corresponding |TS| logging field symbols.
-
-=================== =============
-Netscape Extended-2 Field Symbols
-=================== =============
-``host`` ``chi``
-``usr`` ``caun``
-``[time]`` ``[cqtn]``
-``"req"`` ``"cqtx"``
-``s1`` ``pssc``
-``c1`` ``pscl``
-``s2`` ``sssc``
-``c2`` ``sscl``
-``b1`` ``cqcl``
-``b2`` ``pqcl``
-``h1`` ``cqhl``
-``h2`` ``pshl``
-``h3`` ``pqhl``
-``h4`` ``sshl``
-``xt`` ``tts``
-``route`` ``phr``
-``pfs`` ``cfsc``
-``ss`` ``pfsc``
-``crc`` ``crc``
-=================== =============
-
-This is the equivalent XML configuration for the log above::
-
- <LogFormat>
- <Name = "extended2"/>
- <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>
- %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts> %<phr> %<cfsc> %<pfsc> %<crc>"/>
- </LogFormat>
-
-.. _log-field-slicing:
-
-Log Field Slicing
------------------
-
-It is sometimes desirable to slice a log field to limit the length of a given
-log field's output.
-
-Log Field slicing can be specified as below::
-
- %<field[start:end]>
- %<{field}container[start:end]>
-
-Omitting the slice notation defaults to the entire log field.
-
-Slice notation only applies to a log field that is of type string and can not
-be applied to IPs or timestamp which are converted to string from integer.
-
-The below slice specifiers are allowed.
-
-``[start:end]``
- Log field value from start through end-1
-``[start:]``
- Log field value from start through the rest of the string
-``[:end]``
- Log field value from the beginning through end-1
-``[:]``
- Default - entire Log field
-
-Some examples below ::
-
- '%<cqup>' // the whole characters of <cqup>.
- '%<cqup>[:]' // the whole characters of <cqup>.
- '%<cqup[0:30]>' // the first 30 characters of <cqup>.
- '%<cqup[-10:]>' // the last 10 characters of <cqup>.
- '%<cqup[:-5]>' // everything except the last 5 characters of <cqup>.
-
-.. _admin-monitoring-logging-cache-result-codes:
-
-Cache Result Codes
-==================
-
-The following table describes the cache result codes in Squid and
-Netscape log files.
-
-``TCP_HIT``
- A valid copy of the requested object was in the cache and Traffic
- Server sent the object to the client.
-
-``TCP_MISS``
- The requested object was not in cache, so |TS| retrieved
- the object from the origin server (or a parent proxy) and sent it to
- the client.
-
-``TCP_REFRESH_HIT``
- The object was in the cache, but it was stale. |TS| made an
- ``if-modified-since`` request to the origin server and the
- origin server sent a ``304`` not-modified response. Traffic
- Server sent the cached object to the client.
-
-``TCP_REF_FAIL_HIT``
- The object was in the cache but was stale. |TS| made an
- ``if-modified-since`` request to the origin server but the server
- did not respond. |TS| sent the cached object to the
- client.
-
-``TCP_REFRESH_MISS``
- The object was in the cache but was stale. |TS| made an
- ``if-modified-since`` request to the origin server and the server
- returned a new object. |TS| served the new object to the
- client.
-
-``TCP_CLIENT_REFRESH``
- The client issued a request with a ``no-cache`` header. Traffic
- Server obtained the requested object from the origin server and sent
- a copy to the client. |TS| deleted the previous copy of
- the object from cache.
-
-``TCP_IMS_HIT``
- The client issued an ``if-modified-since`` request and the object
- was in cache and fresher than the IMS date, or an
- ``if-modified-since`` request to the origin server revealed the
- cached object was fresh. |TS| served the cached object to
- the client.
-
-``TCP_IMS_MISS``
- The client issued an
- ``if-modified-since request`` and the object was either not in
- cache or was stale in cache. |TS| sent an
- ``if-modified-since request`` to the origin server and received the
- new object. |TS| sent the updated object to the client.
-
-``TCP_SWAPFAIL``
- The object was in the cache but could not be accessed. The client
- did not receive the object.
-
-``ERR_CLIENT_ABORT``
- The client disconnected before the complete object was sent.
-
-``ERR_CONNECT_FAIL``
- |TS| could not reach the origin server.
-
-``ERR_DNS_FAIL``
- The Domain Name Server (DNS) could not resolve the origin server
- name, or no DNS could be reached.
-
-``ERR_INVALID_REQ``
- The client HTTP request was invalid. (|TS| forwards
- requests with unknown methods to the origin server.)
-
-``ERR_READ_TIMEOUT``
- The origin server did not respond to |TS|'s request within
- the timeout interval.
-
-``ERR_PROXY_DENIED``
- Client service was denied.
-
-``ERR_UNKNOWN``
- The client connected, but subsequently disconnected without sending
- a request.
diff --git a/doc/admin-guide/monitoring/logging/pipes.en.rst b/doc/admin-guide/monitoring/logging/pipes.en.rst
deleted file mode 100644
index d6f99c7..0000000
--- a/doc/admin-guide/monitoring/logging/pipes.en.rst
+++ /dev/null
@@ -1,35 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-.. include:: ../../../common.defs
-
-.. _admin-monitoring-logging-pipes:
-
-ASCII Log Pipes
-***************
-
-In addition to ``ASCII`` and ``BINARY`` file modes for custom log formats, |TS|
-can output log entries in ``ASCII_PIPE`` mode. This mode writes the log entries
-to a UNIX named pipe (a buffer in memory). Other processes may read from this
-named pipe using standard I/O functions.
-
-The advantage of this mode is that |TS| does not need to write the entries to
-disk, which frees disk space and bandwidth for other tasks. When the buffer is
-full, |TS| drops log entries and issues an error message indicating how many
-entries were dropped. Because |TS| only writes complete log entries to the
-pipe, only full records are dropped.
-
diff --git a/doc/admin-guide/monitoring/logging/summary-logs.en.rst b/doc/admin-guide/monitoring/logging/summary-logs.en.rst
deleted file mode 100644
index 4e51b9e..0000000
--- a/doc/admin-guide/monitoring/logging/summary-logs.en.rst
+++ /dev/null
@@ -1,101 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-.. include:: ../../../common.defs
-
-.. _admin-monitoring-logging-summary-logs:
-
-Summary Logs
-************
-
-Due to the speed and efficiency of |TS|, a heavily-loaded node will generate
-many events and the event logs can quickly grow to very large sizes. Using
-SQL-like aggregate operators, you can configure |TS| to create summary log
-files that summarize a set of log entries over a specified period of time. This
-can significantly reduce the size of the log files generated.
-
-To generate a summary log file, create a ``format`` object in the
-custom logging configuration :file:`logging.config` using the SQL-like
-aggregate operators below. You can apply each of these operators to specific
-fields, over a specified interval.
-
-- ``COUNT``
-- ``SUM``
-- ``AVERAGE``
-- ``FIRST``
-- ``LAST``
-
-Creating Summary Log Formats
-============================
-
-To create a summary log file format:
-
-#. Define the format of the log file in :file:`logging.config` as follows:
-
- .. code:: lua
-
- mysummary = format {
- Format = "%<operator(field)> : %<operator(field)>",
- Interval = "n"
- }
-
- Where ``operator`` is one of the five aggregate operators detailed earlier
- and ``field`` is the logging field you want to aggregate. For the interval,
- ``n`` is the interval (in seconds) between summary log entries.
-
- You can specify more than one ``operator`` in the format line. For more
- information, refer to :file:`logging.config`.
-
-#. Run the command :option:`traffic_ctl config reload` to apply the changes.
-
-Mixing Normal Fields and Aggregates
-===================================
-
-It is important to note that you cannot create custom formats for summary logs
-which mix both normal, unadorned fields and aggregate functions. Your custom
-format must be all of one, or all of the other. In other words, the following
-attempt would not work:
-
-.. code:: lua
-
- wontwork = format {
- Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)> : %<cqu>",
- Interval = "10"
- }
-
-Because it attempts to use the last, count, and sum aggregate functions at the
-same time it also includes the bare, unaggregated ``cqu`` field. As each line
-of the summary log may be summarized across many individual events, |TS| would
-have no way of knowing which individual event's ``cqu`` should be emitted to
-the log.
-
-Examples
-========
-
-The following example format generates one entry every 10 seconds. Each entry
-contains the timestamp of the last entry of the interval, a count of the number
-of entries seen within that 10-second interval, and the sum of all bytes sent
-to the client:
-
-
-.. code:: lua
-
- mysummary = format {
- Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>",
- Interval = "10"
- }
-
diff --git a/doc/admin-guide/monitoring/logging/understanding.en.rst b/doc/admin-guide/monitoring/logging/understanding.en.rst
deleted file mode 100644
index c10f67f..0000000
--- a/doc/admin-guide/monitoring/logging/understanding.en.rst
+++ /dev/null
@@ -1,83 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-.. include:: ../../../common.defs
-
-.. _admin-monitoring-logging-understanding:
-
-Understanding |TS| Log Files
-****************************
-
-|TS| records information about every transaction (or request) it processes and
-every error it detects in log files. |TS| keeps three types of log files:
-
-- *Error log files* record information about why a particular
- transaction was in error.
-
-- *Event log files* (also called *access log files*) record
- information about the state of each transaction Traffic Server
- processes.
-
-- *System log files* record system information, including messages
- about the state of Traffic Server and errors/warnings it produces.
- This kind of information might include a note that event log files
- were rolled, a warning that cluster communication timed out, or an
- error indicating that Traffic Server was restarted.
-
- All system information messages are logged with the system-wide
- logging facility :manpage:`syslog` under the daemon facility. The
- :manpage:`syslog.conf(5)` configuration file (stored in the ``/etc`` directory)
- specifies where these messages are logged. A typical location is
- ``/var/log/messages`` (Linux).
-
- The :manpage:`syslog(8)` process works on a system-wide basis, so it serves as
- the single repository for messages from all Traffic Server processes
- (including :program:`traffic_server`, :program:`traffic_manager`, and
- :program:`traffic_cop`).
-
- System information logs observe a static format. Each log entry in
- the log contains information about the date and time the error was
- logged, the hostname of the Traffic Server that reported the error,
- and a description of the error or warning.
-
- Refer to :ref:`admin-monitoring-errors` for a list of the
- messages logged by Traffic Server.
-
-By default, |TS| creates both error and event log files and
-records system information in system log files. You can disable event
-logging and/or error logging by setting the configuration variable
-:ts:cv:`proxy.config.log.logging_enabled` in :file:`records.config`
-to one of the following values:
-
-======= =================================================
-Value Description
-======= =================================================
-``0`` Disable both event and error logging.
-``1`` Enable error logging only.
-``2`` Enable event logging only.
-``3`` Enable both event and error logging.
-======= =================================================
-
-By analyzing the log files, you can determine how many people use the |TS|
-cache, how much information each person requested, what pages are most popular,
-and so on. |TS| supports several standard log file formats, such as Squid and
-Netscape, as well as user-defined custom formats. You can analyze the standard
-format log files with off-the-shelf analysis packages. To help with log file
-analysis, you can separate log files so they contain information specific to
-protocol or hosts. You can also configure |TS| to roll log files automatically
-at specific intervals during the day or when they reach a certain size.
-
diff --git a/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst b/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst
index 9ae39f2..f5705fe 100644
--- a/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst
+++ b/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst
@@ -41,8 +41,8 @@ that is currently left as an exercise for the reader.
================
|TS| provides a very flexible set of logging outputs. Almost any format can be
-constructed. The full range of options is covered in the
-:ref:`admin-monitoring-logging` chapter.
+constructed. The full range of options is covered in the :ref:`admin-logging`
+chapter.
This guide will walk you through using the appropriate filters in Logstash for
the common logging formats in |TS|. If you have constructed your own custom log
@@ -84,7 +84,7 @@ custom log formats for |TS|, you may need to write your own patterns, however.
Squid Compatible
----------------
-The :ref:`admin-logging-format-squid` log format includes, unsurprisingly, a
+The :ref:`admin-logging-examples-squid` log format includes, unsurprisingly, a
few useful fields for proxy servers. Using the following grok pattern will
extract this information from your |TS| logs if you employ the Squid compatible
log format::
@@ -107,7 +107,7 @@ timestamp Date and time of the client request.
timetoserve Time, in seconds, from initial client connection to |TS| until the
last byte has been sent back to client from |TS|.
clientip Client IP address or hostname.
-cachecode :ref:`admin-monitoring-logging-cache-result-codes`.
+cachecode :ref:`admin-logging-crc`.
response HTTP response status code sent by |TS| to the client.
bytes Length, in bytes, of the |TS| response to the client, including
headers.
@@ -122,7 +122,7 @@ contenttype Content type of the response.
Netscape Common
---------------
-If your |TS| instance is already outputting :ref:`admin-logging-format-common`
+If your |TS| instance is already outputting :ref:`admin-logging-examples-netscape`
format logs, then Logstash's ``COMMONAPACHELOG`` pattern will handle your logs
out of the box. Add the following filter block to your Logstash configuration::
@@ -162,7 +162,7 @@ Netscape Extended
-----------------
The following pattern adds to Common Apache to support the additional fields
-found in :ref:`admin-logging-format-extended`::
+found in :ref:`admin-logging-examples-extended`::
filter {
grok {
diff --git a/doc/static/images/admin/logcolat.jpg b/doc/static/images/admin/logcolat.jpg
deleted file mode 100644
index 4b489f3..0000000
Binary files a/doc/static/images/admin/logcolat.jpg and /dev/null differ
--
To stop receiving notification emails like this one, please contact
['"commits@trafficserver.apache.org" <co...@trafficserver.apache.org>'].