You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by gi...@apache.org on 2019/01/09 19:37:39 UTC
[mesos] branch 1.4.x updated (6791ab0 -> 83d1ae6)
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a change to branch 1.4.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 6791ab0 Disabled warnings-as-errors for gRPC build.
new 320a293 Sent SIGKILL to I/O switchboard server as a safeguard.
new 83d1ae6 Fixed the FD leak if containerizer::_launch() failed or discarded.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
src/slave/containerizer/mesos/containerizer.cpp | 54 ++++++++++++------------
src/slave/containerizer/mesos/io/switchboard.cpp | 17 ++++++++
2 files changed, 45 insertions(+), 26 deletions(-)
[mesos] 01/02: Sent SIGKILL to I/O switchboard server as a
safeguard.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.4.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 320a293ad8db41a5061014ffbe6cf6e6f59a8795
Author: Qian Zhang <zh...@gmail.com>
AuthorDate: Mon Jan 7 16:16:12 2019 -0800
Sent SIGKILL to I/O switchboard server as a safeguard.
Review: https://reviews.apache.org/r/69667/
(cherry picked from commit 3478e344fb77d931f6122980c6e94cd3913c441d)
---
src/slave/containerizer/mesos/io/switchboard.cpp | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/slave/containerizer/mesos/io/switchboard.cpp b/src/slave/containerizer/mesos/io/switchboard.cpp
index d36ec79..a02fc98 100644
--- a/src/slave/containerizer/mesos/io/switchboard.cpp
+++ b/src/slave/containerizer/mesos/io/switchboard.cpp
@@ -797,6 +797,23 @@ Future<Nothing> IOSwitchboard::cleanup(
<< " is being destroyed";
os::kill(pid.get(), SIGTERM);
+
+ Clock::timer(Seconds(60), [pid, status, containerId]() {
+ if (status.isPending()) {
+ // If we are here, something really bad must have happened for I/O
+ // switchboard server to not exit after SIGTERM has been sent. We
+ // have seen this happen due to FD leak (see MESOS-9502). We do a
+ // SIGKILL here as a safeguard so that switchboard server forcefully
+ // exits and causes this cleanup feature to be completed, thus
+ // unblocking the container's cleanup.
+ LOG(ERROR) << "Sending SIGKILL to I/O switchboard server (pid: "
+ << pid.get() << ") for container " << containerId
+ << " since the I/O switchboard server did not terminate "
+ << "60 seconds after SIGTERM was sent to it";
+
+ os::kill(pid.get(), SIGKILL);
+ }
+ });
}
});
}
[mesos] 02/02: Fixed the FD leak if containerizer::_launch() failed
or discarded.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.4.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 83d1ae6100421d06dba6887c6c493e103dc90dcf
Author: Andrei Budnik <ab...@mesosphere.com>
AuthorDate: Wed Jan 9 11:09:08 2019 -0800
Fixed the FD leak if containerizer::_launch() failed or discarded.
For the period between IOSB server is up and container process exec,
if the containerizer launch returns failure or discarded, the FD used
for signaling from the container process to the IOSB finish redirect
will be leaked, which would cause the IOSB stuck at `io::redirect`
forever. It would block the containerizer from cleaning up this
container at IOSB.
This issue could be exposed if there are frequent check containers
launch on an agent with heavy loads.
This patch fixes the issue by handling discard of a `launch` future,
so the container IO is cleaned up and therefore all FDs are closed.
Review: https://reviews.apache.org/r/69684/
(cherry picked from commit 6938af6e8edc15b24846adface1eeb98032e3463)
---
src/slave/containerizer/mesos/containerizer.cpp | 54 +++++++++++++------------
1 file changed, 28 insertions(+), 26 deletions(-)
diff --git a/src/slave/containerizer/mesos/containerizer.cpp b/src/slave/containerizer/mesos/containerizer.cpp
index 73334ff..d7db2b8 100644
--- a/src/slave/containerizer/mesos/containerizer.cpp
+++ b/src/slave/containerizer/mesos/containerizer.cpp
@@ -1130,40 +1130,42 @@ Future<bool> MesosContainerizerProcess::launch(
containers_.put(containerId, container);
+ Future<Nothing> _prepare;
+
// We'll first provision the image for the container, and
// then provision the images specified in `volumes` using
// the 'volume/image' isolator.
if (!containerConfig.has_container_info() ||
!containerConfig.container_info().mesos().has_image()) {
- return prepare(containerId, None())
- .then(defer(self(), [this, containerId] () {
- return ioSwitchboard->extractContainerIO(containerId);
- }))
- .then(defer(self(),
- &Self::_launch,
- containerId,
- lambda::_1,
- environment,
- pidCheckpointPath));
- }
-
- container->provisioning = provisioner->provision(
+ _prepare = prepare(containerId, None());
+ } else {
+ container->provisioning = provisioner->provision(
containerId,
containerConfig.container_info().mesos().image());
- return container->provisioning
- .then(defer(self(),
- [=](const ProvisionInfo& provisionInfo) -> Future<bool> {
- return prepare(containerId, provisionInfo)
- .then(defer(self(), [this, containerId] () {
- return ioSwitchboard->extractContainerIO(containerId);
- }))
- .then(defer(self(),
- &Self::_launch,
- containerId,
- lambda::_1,
- environment,
- pidCheckpointPath));
+ _prepare = container->provisioning
+ .then(defer(self(), [=](const ProvisionInfo& provisionInfo) {
+ return prepare(containerId, provisionInfo);
+ }));
+ }
+
+ return _prepare
+ .then(defer(self(), [this, containerId] () {
+ return ioSwitchboard->extractContainerIO(containerId);
+ }))
+ .then(defer(self(), [=](const Option<ContainerIO>& containerIO) {
+ return _launch(containerId, containerIO, environment, pidCheckpointPath);
+ }))
+ .onAny(defer(self(), [this, containerId](
+ const Future<Containerizer::LaunchResult>& future) {
+ // We need to clean up the container IO in the case when IOSwitchboard
+ // process has started, but we have not taken ownership of the container
+ // IO by calling `extractContainerIO()`. This may happen if `launch`
+ // future is discarded by the caller of this method. The container IO
+ // stores FDs in the `FDWrapper` struct, which closes these FDs on its
+ // destruction. Otherwise, IOSwitchboard might get stuck trying to read
+ // leaked FDs.
+ ioSwitchboard->extractContainerIO(containerId);
}));
}