You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by gi...@apache.org on 2019/02/21 00:57:48 UTC
[mesos] branch 1.7.x updated (0350d5f -> 87f0f28)
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a change to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 0350d5f Blacklisted the "ubuntu-4" Jenkins box.
new eb11752 Handled containers which has no checkpointed volumes during recovery.
new 87f0f28 Added MESOS-9507 to 1.7.2 CHANGELOG.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG | 1 +
.../mesos/isolators/docker/volume/isolator.cpp | 27 ++++++++++++++++++++--
2 files changed, 26 insertions(+), 2 deletions(-)
[mesos] 01/02: Handled containers which has no checkpointed volumes
during recovery.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit eb117529c100fa2da7ba2b63020094d975418e95
Author: Qian Zhang <zh...@gmail.com>
AuthorDate: Wed Feb 20 13:41:51 2019 -0800
Handled containers which has no checkpointed volumes during recovery.
There are two cases we need to handle:
1. The container's checkpointed docker volumes file does not exist.
2. The container's checkpointed docker volumes file is empty.
For both of the two cases, in the recovery of `docker/volume` isolator,
we should construct an info object with empty docker volumes for the
container and rely on containerizer or `docker/volume` isolator's
`recover` method to cleanup the container.
Review: https://reviews.apache.org/r/69972/
(cherry picked from commit d174f0bb5120c70f48cfee2bea5b84724493d416)
---
.../mesos/isolators/docker/volume/isolator.cpp | 27 ++++++++++++++++++++--
1 file changed, 25 insertions(+), 2 deletions(-)
diff --git a/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp b/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp
index 24c9fd6..c924dde 100644
--- a/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp
+++ b/src/slave/containerizer/mesos/isolators/docker/volume/isolator.cpp
@@ -266,8 +266,16 @@ Try<Nothing> DockerVolumeIsolatorProcess::_recover(
paths::getVolumesPath(rootDir, containerId.value());
if (!os::exists(volumesPath)) {
- VLOG(1) << "The docker volumes checkpointed at '" << volumesPath
- << "' for container " << containerId << " does not exist";
+ // This could happen if the slave died after creating the container
+ // directory but before it checkpointed anything in it.
+ LOG(WARNING) << "The docker volumes checkpointed at '" << volumesPath
+ << "' for container " << containerId << " does not exist";
+
+ // Construct an info object with empty docker volumes since no docker
+ // volumes are mounted yet for this container, and this container will
+ // be cleaned up by containerizer (as known orphan container) or by
+ // `recover` (as unknown orphan container).
+ infos.put(containerId, Owned<Info>(new Info(hashset<DockerVolume>())));
return Nothing();
}
@@ -279,6 +287,21 @@ Try<Nothing> DockerVolumeIsolatorProcess::_recover(
volumesPath + "': " + read.error());
}
+ if (read->empty()) {
+ // This could happen if the slave is hard rebooted after the file is
+ // created but before the data is synced on disk.
+ LOG(WARNING) << "The docker volumes checkpointed at '" << volumesPath
+ << "' for container " << containerId << " is empty";
+
+ // Construct an info object with empty docker volumes since no docker
+ // volumes are mounted yet for this container, and this container will
+ // be cleaned up by containerizer (as known orphan container) or by
+ // `recover` (as unknown orphan container).
+ infos.put(containerId, Owned<Info>(new Info(hashset<DockerVolume>())));
+
+ return Nothing();
+ }
+
Try<JSON::Object> json = JSON::parse<JSON::Object>(read.get());
if (json.isError()) {
return Error("JSON parse failed: " + json.error());
[mesos] 02/02: Added MESOS-9507 to 1.7.2 CHANGELOG.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 87f0f287ede661795f6d5890d85614da9d03222c
Author: Gilbert Song <so...@gmail.com>
AuthorDate: Wed Feb 20 16:19:59 2019 -0800
Added MESOS-9507 to 1.7.2 CHANGELOG.
(cherry picked from commit 5f233ddbf4c0b807b4ad35081b058846fea8f3f9)
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index 89efa03..cd453d1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ Release Notes - Mesos - Version 1.7.2
** Bug
* [MESOS-8887] - Unreachable tasks are not GC'ed when unreachable agent is GC'ed.
* [MESOS-9210] - Mesos v1 scheduler library does not properly handle SUBSCRIBE retries.
+ * [MESOS-9507] - Agent could not recover due to empty docker volume checkpointed files.
* [MESOS-9517] - SLRP should treat gRPC timeouts as non-terminal errors, instead of reporting OPERATION_FAILED.
* [MESOS-9531] - chown error handling is incorrect in createSandboxDirectory.
* [MESOS-9532] - ResourceOffersTest.ResourceOfferWithMultipleSlaves is flaky.