You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by ji...@apache.org on 2018/08/20 23:35:05 UTC
[mesos] branch 1.5.x updated (35bd5cc -> 8905f29)
This is an automated email from the ASF dual-hosted git repository.
jieyu pushed a change to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 35bd5cc Added MESOS-9116 to the 1.5.2 CHANGELOG.
new d1fd26b Fixed the broken build.
new a0a0f37 Changed default executor tests to not use pipes for synchronization.
new 99fe564 Avoided leaking file descriptors in Mesos containerizer.
new 8905f29 Added MESOS-8917 to 1.5.2 CHANGELOG.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG | 1 +
src/slave/containerizer/mesos/launch.cpp | 58 +++++++-
src/tests/check_tests.cpp | 41 +-----
.../nested_mesos_containerizer_tests.cpp | 162 +++++++--------------
4 files changed, 117 insertions(+), 145 deletions(-)
[mesos] 02/04: Changed default executor tests to not use pipes for
synchronization.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jieyu pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a0a0f379a75f644abd03c5de7cf7b588119f68c8
Author: Benjamin Bannier <be...@mesosphere.io>
AuthorDate: Mon Jun 25 20:08:43 2018 +0200
Changed default executor tests to not use pipes for synchronization.
Some tests of nested container functionality used pipes passed to
launched tasks to detect whether a task has actually started executing
the workload (`TASK_RUNNING` updates might be sent before the task
workload is actually started).
Once we avoid leaking unspecified file descriptors into forked
processes, this test setup will be broken. In this patch we replace
the use of pipes for synchronization with HTTP requests to an actor
running in the tests, or wait on other observable side effects.
Review: https://reviews.apache.org/r/67398/
(cherry picked from commit a666047c9324a0b24b26fa8d89b3fdb73537f44f)
---
src/tests/check_tests.cpp | 41 ++----
.../nested_mesos_containerizer_tests.cpp | 156 ++++++---------------
2 files changed, 54 insertions(+), 143 deletions(-)
diff --git a/src/tests/check_tests.cpp b/src/tests/check_tests.cpp
index 0163e3a..04e576a 100644
--- a/src/tests/check_tests.cpp
+++ b/src/tests/check_tests.cpp
@@ -1971,47 +1971,22 @@ TEST_F_TEMP_DISABLED_ON_WINDOWS(
.WillOnce(FutureArg<1>(&updateCheckResult))
.WillRepeatedly(Return()); // Ignore subsequent updates.
- // Default executor delegates launching both the task and its check to the
- // agent. To avoid a race, we explicitly synchronize.
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
-
const string filename = "nested_inherits_work_dir";
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'echo', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- v1::CommandInfo command;
- command.set_shell(false);
- command.set_value("/bin/bash");
- command.add_arguments("bash");
- command.add_arguments("-c");
- command.add_arguments(
- "touch " + filename + ";echo running >&" +
- stringify(pipes[1]) + ";sleep 1000");
-
- v1::TaskInfo taskInfo = v1::createTask(agentId, resources, command);
+ v1::TaskInfo taskInfo = v1::createTask(
+ agentId,
+ resources,
+ v1::createCommandInfo(
+ strings::format("touch %s; sleep 1000", filename).get()));
v1::CheckInfo* checkInfo = taskInfo.mutable_check();
checkInfo->set_type(v1::CheckInfo::COMMAND);
checkInfo->set_delay_seconds(0);
checkInfo->set_interval_seconds(0);
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'read', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- v1::CommandInfo* checkCommand =
- checkInfo->mutable_command()->mutable_command();
- checkCommand->set_shell(false);
- checkCommand->set_value("/bin/bash");
- checkCommand->add_arguments("bash");
- checkCommand->add_arguments("-c");
- checkCommand->add_arguments(
- "read INPUT <&" + stringify(pipes[0]) + ";ls " + filename);
+ // Wait in a busy loop until the file has been created.
+ checkInfo->mutable_command()->mutable_command()->CopyFrom(
+ v1::createCommandInfo("while [ -f " + filename + "]; do :; done"));
v1::TaskGroupInfo taskGroup;
taskGroup.add_tasks()->CopyFrom(taskInfo);
diff --git a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
index b441486..579e866 100644
--- a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
+++ b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
@@ -14,6 +14,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+#include <sys/stat.h>
#include <sys/wait.h>
#include <map>
@@ -452,23 +453,11 @@ TEST_F(NestedMesosContainerizerTest,
ContainerID containerId;
containerId.set_value(id::UUID::random().toString());
- // Use a pipe to pass parent's MESOS_SANDBOX value to a child container.
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
+ string pipe = path::join(sandbox.get(), "pipe");
+ ASSERT_EQ(0, ::mkfifo(pipe.c_str(), 0700));
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'echo', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- CommandInfo command;
- command.set_shell(false);
- command.set_value("/bin/bash");
- command.add_arguments("bash");
- command.add_arguments("-c");
- command.add_arguments(
- "echo $MESOS_SANDBOX >&" + stringify(pipes[1]) + ";" + "sleep 1000");
+ CommandInfo command =
+ createCommandInfo("echo ${MESOS_SANDBOX} > " + pipe + " ; sleep 1000");
ExecutorInfo executor = createExecutorInfo("executor", command, "cpus:1");
@@ -486,30 +475,16 @@ TEST_F(NestedMesosContainerizerTest,
AWAIT_ASSERT_EQ(Containerizer::LaunchResult::SUCCESS, launch);
- // Wait for the parent container to start running its task
- // before launching a debug container inside it.
- AWAIT_READY(process::io::poll(pipes[0], process::io::READ));
- close(pipes[1]);
-
- // Launch a nested debug container that compares MESOS_SANDBOX
+ // Launch a nested debug container that compares `MESOS_SANDBOX`
// it sees with the one its parent sees.
{
ContainerID nestedContainerId;
nestedContainerId.mutable_parent()->CopyFrom(containerId);
nestedContainerId.set_value(id::UUID::random().toString());
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'read', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- CommandInfo nestedCommand;
- nestedCommand.set_shell(false);
- nestedCommand.set_value("/bin/bash");
- nestedCommand.add_arguments("bash");
- nestedCommand.add_arguments("-c");
- nestedCommand.add_arguments(
- "read PARENT_SANDBOX <&" + stringify(pipes[0]) + ";"
- "[ ${PARENT_SANDBOX} == ${MESOS_SANDBOX} ] && exit 0 || exit 1;");
+ CommandInfo nestedCommand = createCommandInfo(
+ "read PARENT_SANDBOX < " + pipe + ";"
+ "[ ${PARENT_SANDBOX} = ${MESOS_SANDBOX} ] && exit 0 || exit 1;");
Future<Containerizer::LaunchResult> launchNested = containerizer->launch(
nestedContainerId,
@@ -529,8 +504,6 @@ TEST_F(NestedMesosContainerizerTest,
ASSERT_SOME(waitNested.get());
ASSERT_TRUE(waitNested.get()->has_status());
EXPECT_WEXITSTATUS_EQ(0, waitNested.get()->status());
-
- close(pipes[0]);
}
// Destroy the containerizer with all associated containers.
@@ -574,27 +547,15 @@ TEST_F(NestedMesosContainerizerTest,
containerId.set_value(id::UUID::random().toString());
// Use a pipe to synchronize with the top-level container.
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
+ string pipe = path::join(sandbox.get(), "pipe");
+ ASSERT_EQ(0, ::mkfifo(pipe.c_str(), 0700));
const string filename = "nested_inherits_work_dir";
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'echo', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- CommandInfo command;
- command.set_shell(false);
- command.set_value("/bin/bash");
- command.add_arguments("bash");
- command.add_arguments("-c");
- command.add_arguments(
- "touch " + filename + ";echo running >&" +
- stringify(pipes[1]) + ";sleep 1000");
-
- ExecutorInfo executor = createExecutorInfo("executor", command, "cpus:1");
+ ExecutorInfo executor = createExecutorInfo(
+ "executor",
+ "touch " + filename + "; echo running > " + pipe + "; sleep 1000",
+ "cpus:1");
Try<string> directory = environment->mkdtemp();
ASSERT_SOME(directory);
@@ -617,9 +578,9 @@ TEST_F(NestedMesosContainerizerTest,
// Wait for the parent container to start running its task
// before launching a debug container inside it.
- AWAIT_READY(process::io::poll(pipes[0], process::io::READ));
- close(pipes[1]);
- close(pipes[0]);
+ Result<string> read = os::read(pipe);
+ ASSERT_SOME(read);
+ ASSERT_EQ("running\n", read.get());
Future<ContainerStatus> status = containerizer->status(containerId);
AWAIT_READY(status);
@@ -1019,19 +980,17 @@ TEST_F(NestedMesosContainerizerTest,
ASSERT_EQ(1u, offers->size());
// Use a pipe to synchronize with the top-level container.
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
+ string pipe = path::join(sandbox.get(), "pipe");
+ ASSERT_EQ(0, ::mkfifo(pipe.c_str(), 0700));
- // Launch a command task within the `alpine` docker image and
- // synchronize its launch with the launch of a debug container below.
+ // Launch a command task within the `alpine` docker image.
TaskInfo task = createTask(
offers->front().slave_id(),
offers->front().resources(),
- "echo running >&" + stringify(pipes[1]) + ";" + "sleep 1000");
+ "echo running > /tmp/pipe; sleep 1000");
- task.mutable_container()->CopyFrom(createContainerInfo("alpine"));
+ task.mutable_container()->CopyFrom(createContainerInfo(
+ "alpine", {createVolumeHostPath("/tmp", sandbox.get(), Volume::RW)}));
Future<TaskStatus> statusStarting;
Future<TaskStatus> statusRunning;
@@ -1048,12 +1007,11 @@ TEST_F(NestedMesosContainerizerTest,
AWAIT_READY_FOR(statusRunning, Seconds(120));
ASSERT_EQ(TASK_RUNNING, statusRunning->state());
- close(pipes[1]);
-
// Wait for the parent container to start running its task
// before launching a debug container inside it.
- AWAIT_READY(process::io::poll(pipes[0], process::io::READ));
- close(pipes[0]);
+ Result<string> read = os::read(pipe);
+ ASSERT_SOME(read);
+ ASSERT_EQ("running\n", read.get());
ASSERT_TRUE(statusRunning->has_slave_id());
ASSERT_TRUE(statusRunning->has_container_status());
@@ -1503,23 +1461,14 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentExit)
ContainerID containerId;
containerId.set_value(id::UUID::random().toString());
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
-
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'read', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- CommandInfo command;
- command.set_shell(false);
- command.set_value("/bin/bash");
- command.add_arguments("bash");
- command.add_arguments("-c");
- command.add_arguments("read key <&" + stringify(pipes[0]));
+ string pipe = path::join(sandbox.get(), "pipe");
+ ASSERT_EQ(0, ::mkfifo(pipe.c_str(), 0700));
- ExecutorInfo executor = createExecutorInfo("executor", command, "cpus:1");
+ // We launch a blocking `read` after which we return with a non-success code.
+ ExecutorInfo executor = createExecutorInfo(
+ "executor",
+ createCommandInfo("read < " + pipe + " && exit 1"),
+ "cpus:1");
Try<string> directory = environment->mkdtemp();
ASSERT_SOME(directory);
@@ -1530,8 +1479,6 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentExit)
map<string, string>(),
None());
- close(pipes[0]); // We're never going to read.
-
AWAIT_ASSERT_EQ(Containerizer::LaunchResult::SUCCESS, launch);
// Now launch nested container.
@@ -1552,7 +1499,8 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentExit)
Future<Option<ContainerTermination>> nestedWait = containerizer->wait(
nestedContainerId);
- close(pipes[1]); // Force the 'read key' to exit!
+ // Write to the fifo to unblock the `read` in the parent container.
+ os::write(pipe, "");
AWAIT_READY(wait);
ASSERT_SOME(wait.get());
@@ -1597,25 +1545,14 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentSigterm)
ContainerID containerId;
containerId.set_value(id::UUID::random().toString());
- // Use a pipe to synchronize with the top-level container.
- Try<std::array<int_fd, 2>> pipes_ = os::pipe();
- ASSERT_SOME(pipes_);
-
- const std::array<int_fd, 2>& pipes = pipes_.get();
-
- // NOTE: We use a non-shell command here to use 'bash -c' to execute
- // the 'echo', which deals with the file descriptor, because of a bug
- // in ubuntu dash. Multi-digit file descriptor is not supported in
- // ubuntu dash, which executes the shell command.
- CommandInfo command;
- command.set_shell(false);
- command.set_value("/bin/bash");
- command.add_arguments("bash");
- command.add_arguments("-c");
- command.add_arguments(
- "echo running >&" + stringify(pipes[1]) + ";" + "sleep 1000");
+ // Use a fifo to synchronize with the top-level container.
+ string pipe = path::join(sandbox.get(), "pipe");
+ ASSERT_EQ(0, ::mkfifo(pipe.c_str(), 0700));
- ExecutorInfo executor = createExecutorInfo("executor", command, "cpus:1");
+ ExecutorInfo executor = createExecutorInfo(
+ "executor",
+ createCommandInfo("echo running > " + pipe + "; sleep 1000"),
+ "cpus:1");
Try<string> directory = environment->mkdtemp();
ASSERT_SOME(directory);
@@ -1628,8 +1565,6 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentSigterm)
AWAIT_ASSERT_EQ(Containerizer::LaunchResult::SUCCESS, launch);
- close(pipes[1]);
-
// Now launch nested container.
ContainerID nestedContainerId;
nestedContainerId.mutable_parent()->CopyFrom(containerId);
@@ -1654,8 +1589,9 @@ TEST_F(NestedMesosContainerizerTest, ROOT_CGROUPS_ParentSigterm)
// Wait for the parent container to start running its executor
// process before sending it a signal.
- AWAIT_READY(process::io::poll(pipes[0], process::io::READ));
- close(pipes[0]);
+ Result<string> read = os::read(pipe);
+ ASSERT_SOME(read);
+ ASSERT_EQ("running\n", read.get());
ASSERT_EQ(0u, os::kill(status->executor_pid(), SIGTERM));
[mesos] 03/04: Avoided leaking file descriptors in Mesos
containerizer.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jieyu pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 99fe564c2dc0a1ed81f1aac935479e48526f94f7
Author: Benjamin Bannier <be...@mesosphere.io>
AuthorDate: Mon Jun 25 20:08:48 2018 +0200
Avoided leaking file descriptors in Mesos containerizer.
This patch explicitly closes not required file descriptors when
forking a Mesos containerizer instance. We currently only pass on
stdin, stout, and sterr.
While it would in principle be possible to open all file descriptors
with `O_CLOEXEC`, this is not practical as
* `O_CLOEXEC` is not supported on BSDs (not Windows either, but file
descriptor leaks are prevented there in a different way), and
* we might call thirdparty code outside of our control which does not
use e.g., `O_CLOEXEC` either (the case e.g., for leveldb).
Closing all file descriptors after the fork avoids leaks even in above
scenarios.
Review: https://reviews.apache.org/r/67137/
(cherry picked from commit 3f60c9857f8f68c3d2330ff3d03c8aa84b630db7)
---
src/slave/containerizer/mesos/launch.cpp | 58 +++++++++++++++++++++++++++++++-
1 file changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/slave/containerizer/mesos/launch.cpp b/src/slave/containerizer/mesos/launch.cpp
index 2328ea9..78c6488 100644
--- a/src/slave/containerizer/mesos/launch.cpp
+++ b/src/slave/containerizer/mesos/launch.cpp
@@ -34,8 +34,10 @@
#include <stout/adaptor.hpp>
#include <stout/foreach.hpp>
#include <stout/os.hpp>
-#include <stout/protobuf.hpp>
#include <stout/path.hpp>
+#include <stout/protobuf.hpp>
+#include <stout/stringify.hpp>
+#include <stout/try.hpp>
#include <stout/unreachable.hpp>
#include <stout/os/realpath.hpp>
@@ -66,6 +68,7 @@
using std::cerr;
using std::cout;
using std::endl;
+using std::list;
using std::set;
using std::string;
using std::vector;
@@ -465,6 +468,40 @@ static Try<Nothing> enterChroot(const string& rootfs)
}
+// On Windows all new processes create by Mesos go through the
+// `create_process` wrapper which with the completion of MESOS-8926
+// will prevent inadvertent leaks making this code unnecessary there.
+//
+// TODO(bbannier): Consider moving this to stout as e.g., `os::lsof`.
+#ifndef __WINDOWS__
+static Try<hashset<int_fd>> getOpenFileDescriptors()
+{
+ Try<list<string>> fds =
+#if defined(__linux__)
+ os::ls("/proc/self/fd");
+#elif defined(__APPLE__)
+ os::ls("/dev/fd");
+#endif
+
+ if (fds.isError()) {
+ return Error(fds.error());
+ }
+
+ hashset<int_fd> result;
+ foreach (const string& fd, fds.get()) {
+ Try<int_fd> fd_ = numify<int_fd>(fd);
+
+ if (fd_.isError()) {
+ return Error("Could not interpret file descriptor: " + fd_.error());
+ }
+
+ result.insert(fd_.get());
+ }
+
+ return result;
+}
+#endif // __WINDOWS__
+
int MesosContainerizerLaunch::execute()
{
if (flags.help) {
@@ -968,6 +1005,18 @@ int MesosContainerizerLaunch::execute()
}
#ifndef __WINDOWS__
+ // Construct a set of file descriptors to close before `exec`'ing.
+ Try<hashset<int_fd>> fds = getOpenFileDescriptors();
+ CHECK_SOME(fds);
+
+ std::set<int_fd> whitelistedFds = {
+ STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO};
+
+ // Exclude required file descriptors from closing.
+ foreach (int_fd fd, whitelistedFds) {
+ fds->erase(fd);
+ }
+
// If we have `containerStatusFd` set, then we need to fork-exec the
// command we are launching and checkpoint its status on exit. We
// use fork-exec directly (as opposed to `process::subprocess()`) to
@@ -1038,6 +1087,13 @@ int MesosContainerizerLaunch::execute()
os::close(containerStatusFd.get());
::_exit(EXIT_SUCCESS);
}
+
+ // Avoid leaking not required file descriptors into the forked process.
+ foreach (int_fd fd, fds.get()) {
+ // We use the unwrapped `::close` as opposed to `os::close`
+ // since the former is guaranteed to be async signal safe.
+ ::close(fd);
+ }
}
#endif // __WINDOWS__
[mesos] 04/04: Added MESOS-8917 to 1.5.2 CHANGELOG.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jieyu pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 8905f29826710a9d676b80592b97b21e5d7fd2fd
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Mon Aug 20 16:33:48 2018 -0700
Added MESOS-8917 to 1.5.2 CHANGELOG.
(cherry picked from commit 9c4b4d5efcdeaae561c3e8b736d0467662e80f38)
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index 365110e..38e27d2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,7 @@ Release Notes - Mesos - Version 1.5.2 (WIP)
* [MESOS-8830] - Agent gc on old slave sandboxes could empty persistent volume data
* [MESOS-8904] - Master crash when removing quota.
* [MESOS-8906] - `UriDiskProfileAdaptor` fails to update profile selectors.
+ * [MESOS-8917] - Agent leaking file descriptors into forked processes
* [MESOS-8935] - Quota limit "chopping" can lead to cpu-only and memory-only offers.
* [MESOS-8936] - Implement a Random Sorter for offer allocations.
* [MESOS-8942] - Master streaming API does not send (health) check updates for tasks.
[mesos] 01/04: Fixed the broken build.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jieyu pushed a commit to branch 1.5.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit d1fd26b710c8ced56ca4e631b6cb52440305e413
Author: Jie Yu <yu...@gmail.com>
AuthorDate: Mon Aug 20 16:31:21 2018 -0700
Fixed the broken build.
---
src/tests/containerizer/nested_mesos_containerizer_tests.cpp | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
index 80fd95e..b441486 100644
--- a/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
+++ b/src/tests/containerizer/nested_mesos_containerizer_tests.cpp
@@ -1194,7 +1194,11 @@ TEST_F(NestedMesosContainerizerTest,
EXPECT_WEXITSTATUS_EQ(0, wait.get()->status());
Future<Option<ContainerTermination>> termination =
- containerizer->destroy(containerId);
+ containerizer->wait(containerId);
+
+ Future<bool> destroy = containerizer->destroy(containerId);
+ AWAIT_READY(destroy);
+ ASSERT_TRUE(destroy.get());
AWAIT_READY(termination);
ASSERT_SOME(termination.get());