You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Joseph Wu <jo...@mesosphere.io> on 2015/12/15 03:14:51 UTC
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:14 p.m.)
Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
Changes
-------
Finish up the meat of the logic. Migrated ContainerLogger into the containerizer itself.
Summary (updated)
-----------------
Logger Module: Adds the ContainerLogger into the DockerContainerizer.
Bugs: MESOS-4137
https://issues.apache.org/jira/browse/MESOS-4137
Repository: mesos
Description (updated)
-------
The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
Diffs (updated)
-----
src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
Diff: https://reviews.apache.org/r/41294/diff/
Testing (updated)
-------
Tests will be modified and run later in the review chain.
Thanks,
Joseph Wu
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 14, 2015, 7:51 p.m., Timothy Chen wrote:
> > src/slave/containerizer/docker.cpp, line 139
> > <https://reviews.apache.org/r/41294/diff/2/?file=1164053#file1164053line139>
> >
> > Why not let the create also initialize?
This was an artifact of one of the approaches (now discarded) I tried, regarding where to insert the module.
I agree that we can `init` in `create`.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review110388
-----------------------------------------------------------
On Dec. 14, 2015, 6:14 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 6:14 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review110388
-----------------------------------------------------------
src/slave/containerizer/docker.cpp (line 139)
<https://reviews.apache.org/r/41294/#comment170205>
Why not let the create also initialize?
- Timothy Chen
On Dec. 15, 2015, 2:14 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 2:14 a.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 14, 2015, 8:06 p.m., Timothy Chen wrote:
> > src/docker/docker.cpp, line 603
> > <https://reviews.apache.org/r/41294/diff/2/?file=1164051#file1164051line603>
> >
> > Just a general question, what is the reasoning behind allowing the logger to modify the path and argv of the original subprocess?
>
> Joseph Wu wrote:
> A couple of reasons:
>
> * We need the thing-performing-the-loggging to be run as a process separate from the logging module. `Subprocess::IO::PIPE` will not be sufficient since we can't read from the pipe when the agent process falls over.
> * The logging process might need to do something more complicated than redirecting stdout/stderr to a FD.
> * Some logging daemons (wrappers) are launched by prepending the to the argc/argv array. i.e. `logger-daemon <logger options> <original "path"> <original argv>`
We're going to change this in the next iteration. The logger will only have access to `out` and `err`.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review110395
-----------------------------------------------------------
On Dec. 15, 2015, 12:41 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 12:41 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 14, 2015, 8:06 p.m., Timothy Chen wrote:
> > src/docker/docker.cpp, line 603
> > <https://reviews.apache.org/r/41294/diff/2/?file=1164051#file1164051line603>
> >
> > Just a general question, what is the reasoning behind allowing the logger to modify the path and argv of the original subprocess?
A couple of reasons:
* We need the thing-performing-the-loggging to be run as a process separate from the logging module. `Subprocess::IO::PIPE` will not be sufficient since we can't read from the pipe when the agent process falls over.
* The logging process might need to do something more complicated than redirecting stdout/stderr to a FD.
* Some logging daemons (wrappers) are launched by prepending the to the argc/argv array. i.e. `logger-daemon <logger options> <original "path"> <original argv>`
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review110395
-----------------------------------------------------------
On Dec. 14, 2015, 6:14 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 6:14 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review110395
-----------------------------------------------------------
src/docker/docker.cpp (line 593)
<https://reviews.apache.org/r/41294/#comment170226>
Just a general question, what is the reasoning behind allowing the logger to modify the path and argv of the original subprocess?
- Timothy Chen
On Dec. 15, 2015, 2:14 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 2:14 a.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review112255
-----------------------------------------------------------
Ship it!
Ship It!
- Benjamin Hindman
On Dec. 22, 2015, 1:32 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 22, 2015, 1:32 a.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp c769372090513b702daa2def265aedf536565f96
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/
-----------------------------------------------------------
(Updated Dec. 21, 2015, 5:32 p.m.)
Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
Changes
-------
Remove `ContainerLogger` injection from the `Docker` class.
Bugs: MESOS-4137
https://issues.apache.org/jira/browse/MESOS-4137
Repository: mesos
Description
-------
The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
Diffs (updated)
-----
src/docker/docker.hpp c769372090513b702daa2def265aedf536565f96
src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
Diff: https://reviews.apache.org/r/41294/diff/
Testing
-------
Tests will be modified and run later in the review chain.
Thanks,
Joseph Wu
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Dec. 20, 2015, 3:17 p.m., Benjamin Hindman wrote:
> > We shouldn't pass `ContainerLogger` into `Docker` while also keeping it `Owned` inside `DockerContainerizer`. It's confusing to people, we're explicitly breaking the contract of `Owned`.
> >
> > Instead, why not change the `Docker::run` to take `Subprocess::IO` instead of the stdout/stderr paths? Could we make that work?
Note: The updated diff no longer passes the `ContainerLogger` into the `Docker` class.
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review111430
-----------------------------------------------------------
On Dec. 21, 2015, 5:32 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2015, 5:32 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp c769372090513b702daa2def265aedf536565f96
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/#review111430
-----------------------------------------------------------
We shouldn't pass `ContainerLogger` into `Docker` while also keeping it `Owned` inside `DockerContainerizer`. It's confusing to people, we're explicitly breaking the contract of `Owned`.
Instead, why not change the `Docker::run` to take `Subprocess::IO` instead of the stdout/stderr paths? Could we make that work?
- Benjamin Hindman
On Dec. 15, 2015, 8:41 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41294/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 8:41 p.m.)
>
>
> Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
>
>
> Bugs: MESOS-4137
> https://issues.apache.org/jira/browse/MESOS-4137
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
>
> The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
>
>
> Diffs
> -----
>
> src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
> src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
> src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
> src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
>
> Diff: https://reviews.apache.org/r/41294/diff/
>
>
> Testing
> -------
>
> Tests will be modified and run later in the review chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 41294: Logger Module: Adds the ContainerLogger
into the DockerContainerizer.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41294/
-----------------------------------------------------------
(Updated Dec. 15, 2015, 12:41 p.m.)
Review request for mesos, Benjamin Hindman, haosdent huang, and Artem Harutyunyan.
Changes
-------
Moved `initialize` into `create`. Remove `options` in favor of module parameters.
Bugs: MESOS-4137
https://issues.apache.org/jira/browse/MESOS-4137
Repository: mesos
Description
-------
The `DockerContainerizer` creates and initializes its own `ContainerLogger` and passes it into the `Docker` wrapper.
The `ContainerLogger` is used before launching executors, in two two call sites depending on whether the agent is running in a container or not.
Diffs (updated)
-----
src/docker/docker.hpp 33d6fb3e82ff7328ad093648a45546a18ec7b8cb
src/docker/docker.cpp 5dc4667d93b143b54841d85606affe3e4926757a
src/slave/containerizer/docker.hpp 35712f599395b5f5fbc311a37c6e29b33bac0c8e
src/slave/containerizer/docker.cpp 2d2dd4e0df36207c5f3cbb4fe2c50df51c0f3e9e
Diff: https://reviews.apache.org/r/41294/diff/
Testing
-------
Tests will be modified and run later in the review chain.
Thanks,
Joseph Wu