You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by be...@apache.org on 2016/01/25 05:13:53 UTC
[11/11] mesos git commit: Add documentation for logging and
ContainerLogger.
Add documentation for logging and ContainerLogger.
Moves (and updates) logging flags from configuration.md into the new
logging.md.
Add documentation for the ContainerLogger.
Adds documentation about logging in Master/Agent/Framework and the
ContainerLogger.
Review: https://reviews.apache.org/r/42172/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/4543cdf9
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/4543cdf9
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/4543cdf9
Branch: refs/heads/master
Commit: 4543cdf975937f7b4f99b3115c9081a7fc643388
Parents: 1c19b50
Author: Joseph Wu <jo...@mesosphere.io>
Authored: Sun Jan 24 20:12:55 2016 -0800
Committer: Benjamin Hindman <be...@gmail.com>
Committed: Sun Jan 24 20:13:23 2016 -0800
----------------------------------------------------------------------
docs/configuration.md | 56 +--------
docs/home.md | 1 +
docs/logging.md | 283 +++++++++++++++++++++++++++++++++++++++++++++
docs/modules.md | 2 +-
src/logging/flags.cpp | 35 ++++--
5 files changed, 309 insertions(+), 68 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/4543cdf9/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 1c1e1f5..b1ef131 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -33,6 +33,7 @@ be found by running the binary with the flag `--help`, for example
## Master and Slave Options
*These options can be supplied to both masters and slaves.*
+Also see the [common options for logging](logging.md#Internal).
<table class="table table-striped">
<thead>
@@ -69,16 +70,6 @@ be found by running the binary with the flag `--help`, for example
</tr>
<tr>
<td>
- --external_log_file=VALUE
- </td>
- <td>
- Specify the externally managed log file. This file will be
- exposed in the webui and HTTP API. This is useful when using
- stderr logging as the log file is otherwise unknown to Mesos.
- </td>
- </tr>
- <tr>
- <td>
--firewall_rules=VALUE
</td>
<td>
@@ -111,15 +102,6 @@ be found by running the binary with the flag `--help`, for example
</tr>
<tr>
<td>
- --[no-]initialize_driver_logging
- </td>
- <td>
- Whether to automatically initialize Google logging of scheduler
- and/or executor drivers. (default: true)
- </td>
- </tr>
- <tr>
- <td>
--ip=VALUE
</td>
<td>
@@ -139,34 +121,6 @@ be found by running the binary with the flag `--help`, for example
</tr>
<tr>
<td>
- --log_dir=VALUE
- </td>
- <td>
- Location to put log files. (no default, nothing is written to disk unless
- specified; does not affect logging to stderr)
- </td>
- </tr>
- <tr>
- <td>
- --logbufsecs=VALUE
- </td>
- <td>
- How many seconds to buffer log messages for. (default: 0)
- </td>
- </tr>
- <tr>
- <td>
- --logging_level=VALUE
- </td>
- <td>
- Log message at or above this level. Possible values: <code>INFO</code>,
- <code>WARNING</code>, <code>ERROR</code>; if quiet flag is used, this will
- affect just the logs from <code>log_dir</code>, if specified.
- (default: INFO)
- </td>
- </tr>
- <tr>
- <td>
--port=VALUE
</td>
<td>
@@ -175,14 +129,6 @@ be found by running the binary with the flag `--help`, for example
</tr>
<tr>
<td>
- --[no-]quiet
- </td>
- <td>
- Disable logging to stderr. (default: false)
- </td>
- </tr>
- <tr>
- <td>
--[no-]version
</td>
<td>
http://git-wip-us.apache.org/repos/asf/mesos/blob/4543cdf9/docs/home.md
----------------------------------------------------------------------
diff --git a/docs/home.md b/docs/home.md
index ff797fb..fa9c19c 100644
--- a/docs/home.md
+++ b/docs/home.md
@@ -23,6 +23,7 @@ layout: documentation
* [Framework Authentication](/documentation/latest/authentication/)
* [Framework Authorization](/documentation/latest/authorization/)
* [Framework Rate Limiting](/documentation/latest/framework-rate-limiting/)
+* [Logging](/documentation/latest/logging/)
* [High Availability](/documentation/latest/high-availability/) for running multiple masters simultaneously.
* [Operational Guide](/documentation/latest/operational-guide/)
* [Monitoring](/documentation/latest/monitoring/)
http://git-wip-us.apache.org/repos/asf/mesos/blob/4543cdf9/docs/logging.md
----------------------------------------------------------------------
diff --git a/docs/logging.md b/docs/logging.md
new file mode 100644
index 0000000..d79a747
--- /dev/null
+++ b/docs/logging.md
@@ -0,0 +1,283 @@
+---
+layout: documentation
+---
+
+# Logging
+
+Mesos handles the logs of each Mesos component differently depending on the
+degree of control Mesos has over the source code of the component.
+
+Roughly, these categories are:
+
+* [Internal](#Internal) - Master and Agent.
+* [Containers](#Containers) - Executors and Tasks.
+* External - Components launched outside of Mesos, like
+ Frameworks and [ZooKeeper](high-availability.md). These are expected to
+ implement their own logging solution.
+
+## <a name="Internal"></a>Internal
+
+The Mesos Master and Agent use the
+[Google's logging library](https://github.com/google/glog).
+Google logging options that are not explicitly mentioned below can be
+configured via environment variables.
+
+<table class="table table-striped">
+ <thead>
+ <tr>
+ <th width="30%">
+ Flag
+ </th>
+ <th>
+ Explanation
+ </th>
+ </tr>
+ </thead>
+
+ <tr>
+ <td>
+ <code>--quiet</code>
+ </td>
+ <td>
+ Disable logging to stderr.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>--logging_level=VALUE</code>
+ </td>
+ <td>
+ Log messages at or above this level. Defaults to <code>INFO</code>.
+ Possible values:
+ <code>INFO</code>, <code>WARNING</code>, <code>ERROR</code>.
+
+ If <code>--quiet</code> is specified, this will only affect the logs
+ written to <code>--log_dir</code>, if specified.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>--log_dir=VALUE</code>
+ </td>
+ <td>
+ Location to put log files. By default, nothing is written to disk.
+ Does not affect logging to stderr.
+
+ If specified, the log file will appear in the Mesos WebUI.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>--logbufsecs=VALUE</code>
+ </td>
+ <td>
+ Maximum number of seconds that logs may be buffered for.
+ By default, logs are flushed immediately.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>--[no-]initialize_driver_logging</code>
+ </td>
+ <td>
+ Whether the Master/Agent should initialize Google logging for the Mesos
+ scheduler and executor drivers, in same way as described here.
+ The scheduler/executor drivers have separate logs and do not get
+ written to the Master/Agent.
+
+ This option has no effect when using the HTTP scheduler/executor APIs.
+
+ By default, this option is true.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>--external_log_file=VALUE</code>
+ </td>
+ <td>
+ Location of the externally managed log file. Mesos does not write to
+ this file directly and merely exposes it in the WebUI and HTTP API.
+
+ This is only useful when logging to stderr in combination with an external
+ logging mechanism, like syslog or journald.
+
+ This option is meaningless when specified along with <code>--quiet</code>.
+
+ This option takes precedence over <code>--log_dir</code> in the WebUI.
+ However, logs will still be written to the <code>--log_dir</code> if
+ that option is specified.
+ </td>
+ </tr>
+</table>
+
+Both Master and Agent also expose an HTTP endpoint which temporarily toggles
+verbose logging:
+
+```
+POST <ip:port>/logging/toggle?level=[1|2|3]&duration=VALUE
+```
+
+The effect is analogous to setting the `GLOG_v` environment variable prior
+to starting the Master/Agent, except the logging level will revert to the
+original level after the given duration.
+
+## <a name="Containers"></a>Containers
+
+For background, see [the containerizer documentation](containerizer.md).
+
+Mesos does not assume any structured logging for entities running inside
+containers. Instead, Mesos will store the stdout and stderr of containers
+into plain files ("stdout" and "stderr") located inside
+[the sandbox](sandbox.md#where-is-it).
+
+In some cases, the default Container logger behavior of Mesos is not ideal:
+
+* Logging may not be standardized across containers.
+* Logs are not easily aggregated.
+* Log file sizes are not managed. Given enough time, the "stdout" and "stderr"
+ files can fill up the Agent's disk.
+
+## `ContainerLogger` Module
+
+The `ContainerLogger` module was introduced in Mesos 0.27.0 and aims to address
+the shortcomings of the default logging behavior for containers. The module
+can be used to change how Mesos redirects the stdout and stderr of containers.
+
+The [interface for a `ContainerLogger` can be found here](https://github.com/apache/mesos/blob/master/include/mesos/slave/container_logger.hpp).
+
+Mesos comes with two `ContainerLogger` modules:
+
+* The `SandboxContainerLogger` implements the existing logging behavior as
+ a `ContainerLogger`. This is the default behavior.
+* The `LogrotateContainerLogger` addresses the problem of unbounded log file
+ sizes.
+
+### `LogrotateContainerLogger`
+
+The `LogrotateContainerLogger` constrains the total size of a container's
+stdout and stderr files. The module does this by rotating log files based
+on the parameters to the module. When a log file reaches its specified
+maximum size, it is renamed by appending a `.N` to the end of the filename,
+where `N` increments each rotation. Older log files are deleted when the
+specified maximum number of files is reached.
+
+#### Invoking the module
+
+The `LogrotateContainerLogger` can be loaded by specifying the library
+`liblogrotate_container_logger.so` in the
+[`--modules` flag](modules.md#Invoking) when starting the Agent and by
+setting the `--container_logger` Agent flag to
+`org_apache_mesos_LogrotateContainerLogger`.
+
+#### Module parameters
+
+<table class="table table-striped">
+ <thead>
+ <tr>
+ <th width="30%">
+ Key
+ </th>
+ <th>
+ Explanation
+ </th>
+ </tr>
+ </thead>
+
+ <tr>
+ <td>
+ <code>max_stdout_size</code>/<code>max_stderr_size</code>
+ </td>
+ <td>
+ Maximum size, in bytes, of a single stdout/stderr log file.
+ When the size is reached, the file will be rotated.
+
+ Defaults to 10 MB. Minimum size of 1 (memory) page, usually around 4 KB.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>logrotate_stdout_options</code>/
+ <code>logrotate_stderr_options</code>
+ </td>
+ <td>
+ Additional config options to pass into <code>logrotate</code> for stdout.
+ This string will be inserted into a <code>logrotate</code> configuration
+ file. i.e. For "stdout":
+ <pre>
+/path/to/stdout {
+ [logrotate_stdout_options]
+ size [max_stdout_size]
+}</pre>
+ NOTE: The <code>size</code> option will be overriden by this module.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>launcher_dir</code>
+ </td>
+ <td>
+ Directory path of Mesos binaries.
+ The <code>LogrotateContainerLogger</code> will find the
+ <code>mesos-logrotate-logger</code> binary under this directory.
+
+ Defaults to <code>/usr/local/libexec/mesos</code>.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <code>logrotate_path</code>
+ </td>
+ <td>
+ If specified, the <code>LogrotateContainerLogger</code> will use the
+ specified <code>logrotate</code> instead of the system's
+ <code>logrotate</code>. If <code>logrotate</code> is not found, then
+ the module will exit with an error.
+ </td>
+ </tr>
+</table>
+
+#### How it works
+
+1. Every time a container starts up, the `LogrotateContainerLogger`
+ starts up companion subprocesses of the `mesos-logrotate-logger` binary.
+2. The module instructs Mesos to redirect the container's stdout/stderr
+ to the `mesos-logrotate-logger`.
+3. As the container outputs to stdout/stderr, `mesos-logrotate-logger` will
+ pipe the output into the "stdout"/"stderr" files. As the files grow,
+ `mesos-logrotate-logger` will call `logrotate` to keep the files strictly
+ under the configured maximum size.
+4. When the container exits, `mesos-logrotate-logger` will finish logging before
+ exiting as well.
+
+The `LogrotateContainerLogger` is designed to be resilient across Agent
+failover. If the Agent process dies, any instances of `mesos-logrotate-logger`
+will continue to run.
+
+### Writing a Custom `ContainerLogger`
+
+For basics on module writing, see [the modules documentation](modules.md).
+
+There are several caveats to consider when designing a new `ContainerLogger`:
+
+* Logging by the `ContainerLogger` should be resilient to Agent failover.
+ If the Agent process dies (which includes the `ContainerLogger` module),
+ logging should continue. This is usually achieved by using subprocesses.
+* When containers shut down, the `ContainerLogger` is not explicitly notified.
+ Instead, encountering `EOF` in the container's stdout/stderr signifies
+ that the container has exited. This provides a stronger guarantee that the
+ `ContainerLogger` has seen all the logs before exiting itself.
+* The `ContainerLogger` should not assume that containers have been launched
+ with any specific `ContainerLogger`. The Agent may be restarted with a
+ different `ContainerLogger`.
+* Each [containerizer](containerizer.md) running on an Agent uses its own
+ instance of the `ContainerLogger`. This means more than one `ContainerLogger`
+ may be running in a single Agent. However, each Agent will only run a single
+ type of `ContainerLogger`.
http://git-wip-us.apache.org/repos/asf/mesos/blob/4543cdf9/docs/modules.md
----------------------------------------------------------------------
diff --git a/docs/modules.md b/docs/modules.md
index 1aad8e0..8e0249d 100644
--- a/docs/modules.md
+++ b/docs/modules.md
@@ -29,7 +29,7 @@ Finally, modules provide an easy way for third parties to easily extend Mesos
without having to know all the internal details.
-## Invoking Mesos modules
+## <a name="Invoking"></a>Invoking Mesos modules
The command-line flag `--modules` is available for Mesos master, slave, and
tests to specify a list of modules to be loaded and be available to the internal
http://git-wip-us.apache.org/repos/asf/mesos/blob/4543cdf9/src/logging/flags.cpp
----------------------------------------------------------------------
diff --git a/src/logging/flags.cpp b/src/logging/flags.cpp
index b321c28..978d735 100644
--- a/src/logging/flags.cpp
+++ b/src/logging/flags.cpp
@@ -26,33 +26,44 @@ mesos::internal::logging::Flags::Flags()
add(&Flags::logging_level,
"logging_level",
- "Log message at or above this level; possible values: \n"
- "'INFO', 'WARNING', 'ERROR'; if quiet flag is used, this \n"
- "will affect just the logs from log_dir (if specified)",
+ "Log message at or above this level. Defaults to 'INFO'.\n"
+ "Possible values: 'INFO', 'WARNING', 'ERROR'.\n"
+ "If '--quiet' is specified, this will only affect the logs\n"
+ "written to '--log_dir', if specified.",
"INFO");
add(&Flags::log_dir,
"log_dir",
- "Directory path to put log files (no default, nothing\n"
- "is written to disk unless specified;\n"
- "does not affect logging to stderr).\n"
+ "Location to put log files. By default, nothing is written to disk.\n"
+ "Does not affect logging to stderr.\n"
+ "If specified, the log file will appear in the Mesos WebUI."
"NOTE: 3rd party log messages (e.g. ZooKeeper) are\n"
"only written to stderr!");
add(&Flags::logbufsecs,
"logbufsecs",
- "How many seconds to buffer log messages for",
+ "Maximum number of seconds that logs may be buffered for.\n"
+ "By default, logs are flushed immediately.",
0);
add(&Flags::initialize_driver_logging,
"initialize_driver_logging",
- "Whether to automatically initialize Google logging of scheduler\n"
- "and/or executor drivers.",
+ "Whether the Master/Agent should initialize Google logging for the\n"
+ "Mesos scheduler and executor drivers, in same way as described here.\n"
+ "The scheduler/executor drivers have separate logs and do not get\n"
+ "written to the Master/Agent logs.\n\n"
+ "This option has no effect when using the HTTP scheduler/executor APIs.\n"
+ "By default, this option is true.",
true);
add(&Flags::external_log_file,
"external_log_file",
- "Specified the externally managed log file. This file will be\n"
- "exposed in the webui and HTTP api. This is useful when using\n"
- "stderr logging as the log file is otherwise unknown to Mesos.");
+ "Location of the externally managed log file. Mesos does not write to\n"
+ "this file directly and merely exposes it in the WebUI and HTTP API.\n"
+ "This is only useful when logging to stderr in combination with an\n"
+ "external logging mechanism, like syslog or journald.\n\n"
+ "This option is meaningless when specified along with '--quiet'.\n\n"
+ "This option takes precedence over '--log_dir' in the WebUI.\n"
+ "However, logs will still be written to the '--log_dir' if\n"
+ "that option is specified.");
}