You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by gi...@apache.org on 2018/09/01 08:14:17 UTC
[mesos] branch 1.7.x updated (0d5b34c -> a7dff89)
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a change to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 0d5b34c Moved MESOS-9185 to the 1.7.0 CHANGELOG.
new 2b5a619 Made overlay backend destroy more robust.
new 60f6680 Made aufs backend destroy more robust.
new 6546ec9 Made bind backend destroy more robust.
new 4848002 Made copy backend destroy more robust.
new a7dff89 Added MESOS-9196 to 1.7.0 CHANGELOG.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG | 1 +
.../mesos/provisioner/backends/aufs.cpp | 20 +++++++++++++++-----
.../mesos/provisioner/backends/bind.cpp | 7 +++++--
.../mesos/provisioner/backends/copy.cpp | 12 +++++++++---
.../mesos/provisioner/backends/overlay.cpp | 20 +++++++++++++++-----
5 files changed, 45 insertions(+), 15 deletions(-)
[mesos] 01/05: Made overlay backend destroy more robust.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 2b5a61994739062c3debe38448405fda604ba888
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Fri Aug 31 22:29:36 2018 -0700
Made overlay backend destroy more robust.
Use MNT_DETACH for `unmount` so that if there are still processes
holding files or directories in the rootfs (e.g., regular filesystem
indexing), the unmount will still be successful. The kernel will cleanup
the mount when the number of references reach zero. Also, do not return
hard failure if `rmdir` failed. It's also possible that `rmdir` returns
EBUSY. See more details in MESOS-9196.
Review: https://reviews.apache.org/r/68594/
---
.../mesos/provisioner/backends/overlay.cpp | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/src/slave/containerizer/mesos/provisioner/backends/overlay.cpp b/src/slave/containerizer/mesos/provisioner/backends/overlay.cpp
index e32b991..b12b5b2 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/overlay.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/overlay.cpp
@@ -252,8 +252,11 @@ Future<bool> OverlayBackendProcess::destroy(
foreach (const fs::MountInfoTable::Entry& entry, mountTable->entries) {
if (entry.target == rootfs) {
- // NOTE: This would fail if the rootfs is still in use.
- Try<Nothing> unmount = fs::unmount(entry.target);
+ // NOTE: Use MNT_DETACH here so that if there are still
+ // processes holding files or directories in the rootfs, the
+ // unmount will still be successful. The kernel will cleanup the
+ // mount when the number of references reach zero.
+ Try<Nothing> unmount = fs::unmount(entry.target, MNT_DETACH);
if (unmount.isError()) {
return Failure(
"Failed to destroy overlay-mounted rootfs '" + rootfs + "': " +
@@ -262,9 +265,16 @@ Future<bool> OverlayBackendProcess::destroy(
Try<Nothing> rmdir = os::rmdir(rootfs);
if (rmdir.isError()) {
- return Failure(
- "Failed to remove rootfs mount point '" + rootfs + "': " +
- rmdir.error());
+ // NOTE: Due to the use of MNT_DETACH above, it's possible
+ // that `rmdir` will fail with EBUSY if some other mounts in
+ // other mount namespaces are still on this mount point on
+ // some old kernel (https://lwn.net/Articles/570338/). No need
+ // to return a hard failure here because the directory will be
+ // removed later and re-attempted on agent recovery.
+ //
+ // TODO(jieyu): Consider only ignore EBUSY error.
+ LOG(ERROR) << "Failed to remove rootfs mount point "
+ << "'" << rootfs << "': " << rmdir.error();
}
// Clean up tempDir used for image layer links.
[mesos] 05/05: Added MESOS-9196 to 1.7.0 CHANGELOG.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a7dff89f6714b041c243bd93ed39dd9696f79b55
Author: Gilbert Song <so...@gmail.com>
AuthorDate: Sat Sep 1 01:07:09 2018 -0700
Added MESOS-9196 to 1.7.0 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index c29ce3e..246f7d5 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -237,6 +237,7 @@ All Resolved Issues:
* [MESOS-9177] - Mesos master segfaults when responding to /state requests.
* [MESOS-9185] - An attempt to remove or destroy container in composing containerizer leads to segfault.
* [MESOS-9193] - Mesos build fail with Clang 3.5.
+ * [MESOS-9196] - Removing rootfs mounts may fail with EBUSY.
** Epic
* [MESOS-8564] - Port libprocess-tests suites to Windows
[mesos] 02/05: Made aufs backend destroy more robust.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 60f6680854b06c5fe98900d1c17908f4bccbc6ed
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Fri Aug 31 22:29:41 2018 -0700
Made aufs backend destroy more robust.
Use MNT_DETACH for `unmount` so that if there are still processes
holding files or directories in the rootfs (e.g., regular filesystem
indexing), the unmount will still be successful. The kernel will cleanup
the mount when the number of references reach zero. Also, do not return
hard failure if `rmdir` failed. It's also possible that `rmdir` returns
EBUSY. See more details in MESOS-9196.
Review: https://reviews.apache.org/r/68595/
---
.../mesos/provisioner/backends/aufs.cpp | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/src/slave/containerizer/mesos/provisioner/backends/aufs.cpp b/src/slave/containerizer/mesos/provisioner/backends/aufs.cpp
index c2cdd93..2eba552 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/aufs.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/aufs.cpp
@@ -251,8 +251,11 @@ Future<bool> AufsBackendProcess::destroy(
foreach (const fs::MountInfoTable::Entry& entry, mountTable->entries) {
if (entry.target == rootfs) {
- // NOTE: This would fail if the rootfs is still in use.
- Try<Nothing> unmount = fs::unmount(entry.target);
+ // NOTE: Use MNT_DETACH here so that if there are still
+ // processes holding files or directories in the rootfs, the
+ // unmount will still be successful. The kernel will cleanup the
+ // mount when the number of references reach zero.
+ Try<Nothing> unmount = fs::unmount(entry.target, MNT_DETACH);
if (unmount.isError()) {
return Failure(
"Failed to destroy aufs-mounted rootfs '" + rootfs + "': " +
@@ -261,9 +264,16 @@ Future<bool> AufsBackendProcess::destroy(
Try<Nothing> rmdir = os::rmdir(rootfs);
if (rmdir.isError()) {
- return Failure(
- "Failed to remove rootfs mount point '" + rootfs + "': " +
- rmdir.error());
+ // NOTE: Due to the use of MNT_DETACH above, it's possible
+ // that `rmdir` will fail with EBUSY if some other mounts in
+ // other mount namespaces are still on this mount point on
+ // some old kernel (https://lwn.net/Articles/570338/). No need
+ // to return a hard failure here because the directory will be
+ // removed later and re-attempted on agent recovery.
+ //
+ // TODO(jieyu): Consider only ignore EBUSY error.
+ LOG(ERROR) << "Failed to remove rootfs mount point "
+ << "'" << rootfs << "': " << rmdir.error();
}
// Clean up tempDir used for image layer links.
[mesos] 04/05: Made copy backend destroy more robust.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 4848002428e466435b4bb43ad30468dc2a34da5c
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Fri Aug 31 22:29:48 2018 -0700
Made copy backend destroy more robust.
Do not return hard failure if `rm -rf` failed. It's also possible that
it fails if some other processes are accessing the files there. The
cleanup will be re-attempted during provisioner destroy and agent
recovery.
Review: https://reviews.apache.org/r/68597/
---
src/slave/containerizer/mesos/provisioner/backends/copy.cpp | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
index d3eb6b4..a3237a3 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/copy.cpp
@@ -326,9 +326,15 @@ Future<bool> CopyBackendProcess::destroy(const string& rootfs)
.then([](const Option<int>& status) -> Future<bool> {
if (status.isNone()) {
return Failure("Failed to reap subprocess to destroy rootfs");
- } else if (status.get() != 0) {
- return Failure("Failed to destroy rootfs, exit status: " +
- WSTRINGIFY(status.get()));
+ }
+
+ if (status.get() != 0) {
+ // It's possible that `rm -rf` will fail if some other
+ // programs are accessing the files. No need to return a hard
+ // failure here because the directory will be removed later
+ // and re-attempted on agent recovery.
+ LOG(ERROR) << "Failed to destroy rootfs, exit status: "
+ << WSTRINGIFY(status.get());
}
return true;
[mesos] 03/05: Made bind backend destroy more robust.
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gilbert pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 6546ec98617e21c15cb9b7d4af20f811b716092b
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Fri Aug 31 22:29:45 2018 -0700
Made bind backend destroy more robust.
Use MNT_DETACH for `unmount` so that if there are still processes
holding files or directories in the rootfs (e.g., regular filesystem
indexing), the unmount will still be successful. The kernel will cleanup
the mount when the number of references reach zero. See more details in
MESOS-9196.
Review: https://reviews.apache.org/r/68596/
---
src/slave/containerizer/mesos/provisioner/backends/bind.cpp | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/src/slave/containerizer/mesos/provisioner/backends/bind.cpp b/src/slave/containerizer/mesos/provisioner/backends/bind.cpp
index 6ad97f9..7d564dc 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/bind.cpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/bind.cpp
@@ -195,8 +195,11 @@ Future<bool> BindBackendProcess::destroy(const string& rootfs)
// to check `strings::startsWith(entry.target, rootfs)` here to
// unmount all nested mounts.
if (entry.target == rootfs) {
- // NOTE: This would fail if the rootfs is still in use.
- Try<Nothing> unmount = fs::unmount(entry.target);
+ // NOTE: Use MNT_DETACH here so that if there are still
+ // processes holding files or directories in the rootfs, the
+ // unmount will still be successful. The kernel will cleanup the
+ // mount when the number of references reach zero.
+ Try<Nothing> unmount = fs::unmount(entry.target, MNT_DETACH);
if (unmount.isError()) {
return Failure(
"Failed to destroy bind-mounted rootfs '" + rootfs + "': " +