You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2020/05/14 04:12:38 UTC
[impala] 03/04: IMPALA-9541: [DOCS] add steps to dynamically change
log levels
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 592ed10a2192cfe87cd05f438028a5293ecadbda
Author: Shajini Thayasingh <st...@cloudera.com>
AuthorDate: Wed May 6 11:58:08 2020 -0700
IMPALA-9541: [DOCS] add steps to dynamically change log levels
added a new section to change log levels dynamically, made review changes
also elaborated the Glog format, remove the word PROCESS
Change-Id: Ic72fbfe3218530f5a1ffa0159ce714e411401bcd
Reviewed-on: http://gerrit.cloudera.org:8080/15867
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/topics/impala_logging.xml | 424 +++++++++++++++++------------------------
1 file changed, 172 insertions(+), 252 deletions(-)
diff --git a/docs/topics/impala_logging.xml b/docs/topics/impala_logging.xml
index ec87e4a..8c20a5e 100644
--- a/docs/topics/impala_logging.xml
+++ b/docs/topics/impala_logging.xml
@@ -41,55 +41,35 @@ under the License.
<conbody>
- <p>
- The Impala logs record information about:
- </p>
+ <p> The Impala logs record information about: </p>
<ul>
- <li>
- Any errors Impala encountered. If Impala experienced a serious error during startup, you
- must diagnose and troubleshoot that problem before you can do anything further with
- Impala.
- </li>
-
- <li>
- How Impala is configured.
- </li>
-
- <li>
- Jobs Impala has completed.
- </li>
+ <li> Any errors Impala encountered. If Impala experienced a serious error during startup, you
+ must diagnose and troubleshoot that problem before you can do anything further with Impala. </li>
+
+ <li> How Impala is configured. </li>
+
+ <li> Jobs Impala has completed. </li>
</ul>
<note>
- <p>
- Formerly, the logs contained the query profile for each query, showing low-level details
+ <p> Formerly, the logs contained the query profile for each query, showing low-level details
of how the work is distributed among nodes and how intermediate and final results are
transmitted across the network. To save space, those query profiles are now stored in
- zlib-compressed files in <filepath>/var/log/impala/profiles</filepath>. You can access
- them through the Impala web user interface. For example, at
- <codeph>http://<varname>impalad-node-hostname</varname>:25000/queries</codeph>, each
+ zlib-compressed files in <filepath>/var/log/impala/profiles</filepath>. You can access them
+ through the Impala web user interface. For example, at
+ <codeph>http://<varname>impalad-node-hostname</varname>:25000/queries</codeph>, each
query is followed by a <codeph>Profile</codeph> link leading to a page showing extensive
- analytical data for the query execution.
- </p>
-
- <p rev="1.1.1">
- The auditing feature introduced in Impala 1.1.1 produces a separate set of audit log
- files when enabled. See <xref href="impala_auditing.xml#auditing"/> for details.
- </p>
-
- <p rev="2.9.0 IMPALA-4431">
- In <keyword keyref="impala29_full"/> and higher, you can control how many audit event
- log files are kept on each host through the
- <codeph>‑‑max_audit_event_log_files</codeph> startup option for the
- <cmdname>impalad</cmdname> daemon, similar to the
- <codeph>‑‑max_log_files</codeph> option for regular log files.
- </p>
-
- <p rev="2.2.0">
- The lineage feature introduced in Impala 2.2.0 produces a separate lineage log file when
- enabled. See <xref href="impala_lineage.xml#lineage"/> for details.
- </p>
+ analytical data for the query execution. </p>
+ <p rev="1.1.1"> The auditing feature introduced in Impala 1.1.1 produces a separate set of
+ audit log files when enabled. See <xref href="impala_auditing.xml#auditing"/> for details. </p>
+ <p rev="2.9.0 IMPALA-4431"> In <keyword keyref="impala29_full"/> and higher, you can control
+ how many audit event log files are kept on each host through the
+ <codeph>‑‑max_audit_event_log_files</codeph> startup option for the
+ <cmdname>impalad</cmdname> daemon, similar to the
+ <codeph>‑‑max_log_files</codeph> option for regular log files. </p>
+ <p rev="2.2.0"> The lineage feature introduced in Impala 2.2.0 produces a separate lineage log
+ file when enabled. See <xref href="impala_lineage.xml#lineage"/> for details. </p>
</note>
<p outputclass="toc inpage"/>
@@ -103,77 +83,55 @@ under the License.
<conbody>
<ul>
- <li>
- By default, the log files are under the directory
- <filepath>/var/log/impala</filepath>. To change log file locations, modify the
- defaults file described in <xref href="impala_processes.xml#processes"/>.
- </li>
+ <li> By default, the log files are under the directory <filepath>/var/log/impala</filepath>.
+ To change log file locations, modify the defaults file described in <xref
+ href="impala_processes.xml#processes"/>. </li>
- <li>
- The significant files for the <codeph>impalad</codeph> process are
- <filepath>impalad.INFO</filepath>, <filepath>impalad.WARNING</filepath>, and
- <filepath>impalad.ERROR</filepath>. You might also see a file
- <filepath>impalad.FATAL</filepath>, although this is only present in rare conditions.
- </li>
+ <li> The significant files for the <codeph>impalad</codeph> process are
+ <filepath>impalad.INFO</filepath>, <filepath>impalad.WARNING</filepath>, and
+ <filepath>impalad.ERROR</filepath>. You might also see a file
+ <filepath>impalad.FATAL</filepath>, although this is only present in rare conditions. </li>
- <li>
- The significant files for the <codeph>statestored</codeph> process are
- <filepath>statestored.INFO</filepath>, <filepath>statestored.WARNING</filepath>, and
- <filepath>statestored.ERROR</filepath>. You might also see a file
- <filepath>statestored.FATAL</filepath>, although this is only present in rare
- conditions.
- </li>
+ <li> The significant files for the <codeph>statestored</codeph> process are
+ <filepath>statestored.INFO</filepath>, <filepath>statestored.WARNING</filepath>, and
+ <filepath>statestored.ERROR</filepath>. You might also see a file
+ <filepath>statestored.FATAL</filepath>, although this is only present in rare
+ conditions. </li>
- <li rev="1.2">
- The significant files for the <codeph>catalogd</codeph> process are
- <filepath>catalogd.INFO</filepath>, <filepath>catalogd.WARNING</filepath>, and
- <filepath>catalogd.ERROR</filepath>. You might also see a file
- <filepath>catalogd.FATAL</filepath>, although this is only present in rare conditions.
- </li>
+ <li rev="1.2"> The significant files for the <codeph>catalogd</codeph> process are
+ <filepath>catalogd.INFO</filepath>, <filepath>catalogd.WARNING</filepath>, and
+ <filepath>catalogd.ERROR</filepath>. You might also see a file
+ <filepath>catalogd.FATAL</filepath>, although this is only present in rare conditions. </li>
- <li>
- Examine the <codeph>.INFO</codeph> files to see configuration settings for the
- processes.
- </li>
+ <li> Examine the <codeph>.INFO</codeph> files to see configuration settings for the
+ processes. </li>
- <li>
- Examine the <codeph>.WARNING</codeph> files to see all kinds of problem information,
- including such things as suboptimal settings and also serious runtime errors.
- </li>
+ <li> Examine the <codeph>.WARNING</codeph> files to see all kinds of problem information,
+ including such things as suboptimal settings and also serious runtime errors. </li>
- <li>
- Examine the <codeph>.ERROR</codeph> and/or <codeph>.FATAL</codeph> files to see only
+ <li> Examine the <codeph>.ERROR</codeph> and/or <codeph>.FATAL</codeph> files to see only
the most serious errors, if the processes crash, or queries fail to complete. These
- messages are also in the <codeph>.WARNING</codeph> file.
- </li>
+ messages are also in the <codeph>.WARNING</codeph> file. </li>
- <li>
- A new set of log files is produced each time the associated daemon is restarted. These
+ <li> A new set of log files is produced each time the associated daemon is restarted. These
log files have long names including a timestamp. The <codeph>.INFO</codeph>,
- <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph> files are physically
- represented as symbolic links to the latest applicable log files.
- </li>
+ <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph> files are physically represented
+ as symbolic links to the latest applicable log files. </li>
</ul>
- <p>
- Impala stores information using the <codeph>glog_v</codeph> logging system. You will see
- some messages referring to C++ file names. Logging is affected by:
- </p>
+ <p> Impala stores information using the <codeph>glog_v</codeph> logging system. You will see
+ some messages referring to C++ file names. Logging is affected by: </p>
<ul>
- <li>
- The <codeph>GLOG_v</codeph> environment variable specifies which types of messages are
- logged. See <xref href="#log_levels"/> for details.
- </li>
-
- <li>
- The <codeph>‑‑logbuflevel</codeph> startup flag for the
- <cmdname>impalad</cmdname> daemon specifies how often the log information is written
- to disk. The default is 0, meaning that the log is immediately flushed to disk when
- Impala outputs an important messages such as a warning or an error, but less important
- messages such as informational ones are buffered in memory rather than being flushed
- to disk immediately.
- </li>
+ <li> The <codeph>GLOG_v</codeph> environment variable specifies which types of messages are
+ logged. See <xref href="#log_levels"/> for details. </li>
+
+ <li> The <codeph>‑‑logbuflevel</codeph> startup flag for the
+ <cmdname>impalad</cmdname> daemon specifies how often the log information is written to
+ disk. The default is 0, meaning that the log is immediately flushed to disk when Impala
+ outputs an important messages such as a warning or an error, but less important messages
+ such as informational ones are buffered in memory rather than being flushed to disk
+ immediately. </li>
</ul>
</conbody>
@@ -192,92 +150,99 @@ under the License.
<conbody>
- <p>
- Impala periodically switches the physical files representing the current log files,
- after which it is safe to remove the old files if they are no longer needed.
- </p>
+ <p> Impala periodically switches the physical files representing the current log files, after
+ which it is safe to remove the old files if they are no longer needed. </p>
- <p>
- Impala can automatically remove older unneeded log files, a feature known as <term>log
- rotation</term>.
- </p>
+ <p> Impala can automatically remove older unneeded log files, a feature known as <term>log
+ rotation</term>. </p>
- <p>
- In Impala 2.2 and higher, the <codeph>‑‑max_log_files</codeph> configuration
- option specifies how many log files to keep at each severity level
- (<codeph>INFO</codeph>, <codeph>WARNING</codeph>, <codeph>ERROR</codeph>, and
- <codeph>FATAL</codeph>). You can specify an appropriate setting for each Impala-related
- daemon (<cmdname>impalad</cmdname>, <cmdname>statestored</cmdname>, and
- <cmdname>catalogd</cmdname>).
- <ul>
- <li>
- A value of 0 preserves all log files, in which case you would set up set up manual
- log rotation using your Linux tool or technique of choice.
- </li>
-
- <li>
- A value of 1 preserves only the very latest log file.
- </li>
-
- <li>
- The default value is 10.
- </li>
+ <p> In Impala 2.2 and higher, the <codeph>‑‑max_log_files</codeph> configuration
+ option specifies how many log files to keep at each severity level (<codeph>INFO</codeph>,
+ <codeph>WARNING</codeph>, <codeph>ERROR</codeph>, and <codeph>FATAL</codeph>). You can
+ specify an appropriate setting for each Impala-related daemon (<cmdname>impalad</cmdname>,
+ <cmdname>statestored</cmdname>, and <cmdname>catalogd</cmdname>). <ul>
+ <li> A value of 0 preserves all log files, in which case you would set up set up manual
+ log rotation using your Linux tool or technique of choice. </li>
+
+ <li> A value of 1 preserves only the very latest log file. </li>
+
+ <li> The default value is 10. </li>
</ul>
</p>
- <p>
- Impala checks to see if any old logs need to be removed based on the interval specified
- in the <codeph>‑‑logbufsecs</codeph> setting, every 5 seconds by default.
- </p>
+ <p> Impala checks to see if any old logs need to be removed based on the interval specified in
+ the <codeph>‑‑logbufsecs</codeph> setting, every 5 seconds by default. </p>
- <p>
- For some log levels, Impala logs are first temporarily buffered in memory and only
- written to disk periodically. The <codeph>‑‑logbufsecs</codeph> setting
- controls the maximum time that log messages are buffered for. For example, with the
- default value of 5 seconds, there may be up to a 5 second delay before a logged message
- shows up in the log file.
- </p>
+ <p> For some log levels, Impala logs are first temporarily buffered in memory and only written
+ to disk periodically. The <codeph>‑‑logbufsecs</codeph> setting controls the
+ maximum time that log messages are buffered for. For example, with the default value of 5
+ seconds, there may be up to a 5 second delay before a logged message shows up in the log
+ file. </p>
- <p>
- It is not recommended that you set <codeph>‑‑logbufsecs</codeph> to 0 as the
- setting makes the Impala daemon to spin in the thread that tries to delete old log
- files.
- </p>
+ <p> It is not recommended that you set <codeph>‑‑logbufsecs</codeph> to 0 as the
+ setting makes the Impala daemon to spin in the thread that tries to delete old log files. </p>
</conbody>
</concept>
+ <concept id="dynamic_log_levels">
+ <title>Changing Log Levels Dynamically</title>
+ <conbody>
+ <p>For debugging purposes you may be adjusting the logging configuration for Catalog and
+ impalad servers. This required restarting the services. Impala supports adjusting the log
+ levels dynamically without the need to restart the server. There is a
+ <codeph>/log_level</codeph> tab in the debug page of all Impala servers. You can query the
+ <codeph>log4j</codeph> log level of <codeph>root</codeph> or
+ <codeph>org.apache.impala</codeph> by using the <codeph>Get Java Log Level</codeph>
+ button. Also you can change the <codeph>vlog/log4j</codeph> levels to any supported levels
+ of logging. You can select the log level using the <codeph>LOG LEVEL</codeph> drop down box.
+ You also have an option to restore the log levels to their original configuration by using
+ the <codeph>RESET</codeph> button.</p>
+ <p>Here is the format of a Glog:</p>
+ <codeblock>${level}${month}${day} HH:MM:SS.${us} ${thread-id} ${source-file}:${line}] ${query-id}] ${message}</codeblock>
+ <p>where</p>
+ <ul id="ul_yqb_ynv_plb">
+ <li>${level} — Log Levels; displays the levels as <codeph>I</codeph> for
+ <codeph>INFO</codeph>, <codeph>W</codeph> for <codeph>WARNING</codeph>,
+ <codeph>E</codeph> for <codeph>ERROR</codeph>, <codeph>F</codeph> for
+ <codeph>FATAL.</codeph>
+ </li>
+ <li>${month}${day} — Month and Date. </li>
+ <li>HH:MM:SS — Hours, Minutes, Seconds. </li>
+ <li>${us} — Microseconds. </li>
+ <li>${thread-id} — TID of the thread. </li>
+ <li>${source-file}:${line}] — File name and line number. </li>
+ <li>${query-id}] — An unique id for each and every query that is run in Impala. </li>
+ <li>${message} — Actual log message.</li>
+ </ul>
+ </conbody>
+ </concept>
+
<concept id="logs_debug">
<title>Reviewing Impala Logs</title>
<conbody>
- <p> By default, the Impala log is stored at
- <codeph>/var/log/impalad/</codeph>. The most comprehensive log,
- showing informational, warning, and error messages, is in the file name
- <filepath>impalad.INFO</filepath>. View log file contents by using the
- web interface or by examining the contents of the log file. (When you
- examine the logs through the file system, you can troubleshoot problems
- by reading the <filepath>impalad.WARNING</filepath> and/or
- <filepath>impalad.ERROR</filepath> files, which contain the subsets of
- messages indicating potential problems.) </p>
+ <p> By default, the Impala log is stored at <codeph>/var/log/impalad/</codeph>. The most
+ comprehensive log, showing informational, warning, and error messages, is in the file name
+ <filepath>impalad.INFO</filepath>. View log file contents by using the web interface or by
+ examining the contents of the log file. (When you examine the logs through the file system,
+ you can troubleshoot problems by reading the <filepath>impalad.WARNING</filepath> and/or
+ <filepath>impalad.ERROR</filepath> files, which contain the subsets of messages indicating
+ potential problems.) </p>
<note>
- <p>
- The web interface limits the amount of logging information displayed. To view every
- log entry, access the log files directly through the file system.
- </p>
+ <p> The web interface limits the amount of logging information displayed. To view every log
+ entry, access the log files directly through the file system. </p>
</note>
- <p>
- You can view the contents of the <codeph>impalad.INFO</codeph> log file in the file
+ <p> You can view the contents of the <codeph>impalad.INFO</codeph> log file in the file
system. With the default configuration settings, the start of the log file appears as
- follows:
- </p>
+ follows: </p>
-<codeblock>[user@example impalad]$ pwd
+ <codeblock>[user@example impalad]$ pwd
/var/log/impalad
[user@example impalad]$ more impalad.INFO
Log file created at: 2013/01/07 08:42:12
@@ -293,10 +258,8 @@ I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug we
--classpath=
--hostname=impala.example.com</codeblock>
- <note>
- The preceding example shows only a small part of the log file. Impala log files are
- often several megabytes in size.
- </note>
+ <note> The preceding example shows only a small part of the log file. Impala log files are
+ often several megabytes in size. </note>
</conbody>
@@ -308,31 +271,19 @@ I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug we
<conbody>
- <p>
- The logs store information about Impala startup options. This information appears once
- for each time Impala is started and may include:
- </p>
+ <p> The logs store information about Impala startup options. This information appears once for
+ each time Impala is started and may include: </p>
<ul>
- <li>
- Machine name.
- </li>
+ <li> Machine name. </li>
- <li>
- Impala version number.
- </li>
+ <li> Impala version number. </li>
- <li>
- Flags used to start Impala.
- </li>
+ <li> Flags used to start Impala. </li>
- <li>
- CPU information.
- </li>
+ <li> CPU information. </li>
- <li>
- The number of available disks.
- </li>
+ <li> The number of available disks. </li>
</ul>
</conbody>
@@ -345,65 +296,40 @@ I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug we
<conbody>
- <p>
- Impala uses the GLOG system, which supports three logging levels. You can adjust logging
- levels by exporting variable settings. To change logging settings manually, use a
- command similar to the following on each node before starting <codeph>impalad</codeph>:
- </p>
+ <p> Impala uses the GLOG system, which supports three logging levels. You can adjust logging
+ levels by exporting variable settings. To change logging settings manually, use a command
+ similar to the following on each node before starting <codeph>impalad</codeph>: </p>
-<codeblock>export GLOG_v=1</codeblock>
+ <codeblock>export GLOG_v=1</codeblock>
- <note>
- For performance reasons, do not enable the most verbose logging level of 3 unless there
- is no other alternative for troubleshooting.
- </note>
+ <note> For performance reasons, do not enable the most verbose logging level of 3 unless there
+ is no other alternative for troubleshooting. </note>
- <p>
- For more information on how to configure GLOG, including how to set variable logging
- levels for different system components, see <xref keyref="glog.html">documentation for
- the glog project on github</xref>.
- </p>
+ <p> For more information on how to configure GLOG, including how to set variable logging
+ levels for different system components, see <xref keyref="glog.html">documentation for the
+ glog project on github</xref>. </p>
<section id="loglevels_details">
-
<title>Understanding What is Logged at Different Logging Levels</title>
-
- <p>
- As logging levels increase, the categories of information logged are cumulative. For
- example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional
- information.
- </p>
-
- <p>
- Increasing logging levels imposes performance overhead and increases log size. Where
+ <p> As logging levels increase, the categories of information logged are cumulative. For
+ example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional information. </p>
+ <p> Increasing logging levels imposes performance overhead and increases log size. Where
practical, use GLOG_v=1 for most cases: this level has minimal performance impact but
- still captures useful troubleshooting information.
- </p>
-
- <p>
- Additional information logged at each level is as follows:
- </p>
-
+ still captures useful troubleshooting information. </p>
+ <p> Additional information logged at each level is as follows: </p>
<ul>
- <li>
- GLOG_v=1 - The default level. Logs information about each connection and query that
- is initiated to an <codeph>impalad</codeph> instance, including runtime profiles.
- </li>
+ <li> GLOG_v=1 - The default level. Logs information about each connection and query that
+ is initiated to an <codeph>impalad</codeph> instance, including runtime profiles. </li>
- <li>
- GLOG_v=2 - Everything from the previous level plus information for each RPC
+ <li> GLOG_v=2 - Everything from the previous level plus information for each RPC
initiated. This level also records query execution progress information, including
- details on each file that is read.
- </li>
-
- <li>
- GLOG_v=3 - Everything from the previous level plus logging of every row that is
- read. This level is only applicable for the most serious troubleshooting and tuning
- scenarios, because it can produce exceptionally large and detailed log files,
- potentially leading to its own set of performance and capacity problems.
- </li>
- </ul>
+ details on each file that is read. </li>
+ <li> GLOG_v=3 - Everything from the previous level plus logging of every row that is read.
+ This level is only applicable for the most serious troubleshooting and tuning scenarios,
+ because it can produce exceptionally large and detailed log files, potentially leading
+ to its own set of performance and capacity problems. </li>
+ </ul>
</section>
</conbody>
@@ -423,28 +349,22 @@ I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug we
<conbody>
<p>
- <term>Log redaction</term> is a security feature that prevents sensitive
- information from being displayed in locations used by administrators for
- monitoring and troubleshooting, such as log files and the Impala debug
- web user interface. You configure regular expressions that match
- sensitive types of information processed by your system, such as credit
- card numbers or tax IDs, and literals matching these patterns are
- obfuscated wherever they would normally be recorded in log files or
- displayed in administration or debugging user interfaces. </p>
-
- <p>
- In a security context, the log redaction feature is complementary to the Sentry
+ <term>Log redaction</term> is a security feature that prevents sensitive information from
+ being displayed in locations used by administrators for monitoring and troubleshooting, such
+ as log files and the Impala debug web user interface. You configure regular expressions that
+ match sensitive types of information processed by your system, such as credit card numbers
+ or tax IDs, and literals matching these patterns are obfuscated wherever they would normally
+ be recorded in log files or displayed in administration or debugging user interfaces. </p>
+
+ <p> In a security context, the log redaction feature is complementary to the Sentry
authorization framework. Sentry prevents unauthorized users from being able to directly
- access table data. Redaction prevents administrators or support personnel from seeing
- the smaller amounts of sensitive or personally identifying information (PII) that might
- appear in queries issued by those authorized users.
- </p>
+ access table data. Redaction prevents administrators or support personnel from seeing the
+ smaller amounts of sensitive or personally identifying information (PII) that might appear
+ in queries issued by those authorized users. </p>
- <p>
- See <xref keyref="sg_redaction"/> for details about how to enable this feature and set
- up the regular expressions to detect and redact sensitive information within SQL
- statement text.
- </p>
+ <p> See <xref keyref="sg_redaction"/> for details about how to enable this feature and set up
+ the regular expressions to detect and redact sensitive information within SQL statement
+ text. </p>
</conbody>