You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by ji...@apache.org on 2016/08/03 00:02:06 UTC
[1/6] mesos git commit: Deprecated the 'rootfs' proto field in
ContainerLaunchInfo.
Repository: mesos
Updated Branches:
refs/heads/master a2541e6b4 -> f0cf8cc37
Deprecated the 'rootfs' proto field in ContainerLaunchInfo.
Review: https://reviews.apache.org/r/50718/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/1cd36028
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/1cd36028
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/1cd36028
Branch: refs/heads/master
Commit: 1cd360282545e14f98f88ee30dee0d31f2aeb7a3
Parents: a2541e6
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:49:40 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:49:40 2016 -0700
----------------------------------------------------------------------
include/mesos/slave/isolator.proto | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/1cd36028/include/mesos/slave/isolator.proto
----------------------------------------------------------------------
diff --git a/include/mesos/slave/isolator.proto b/include/mesos/slave/isolator.proto
index a971a58..e945514 100644
--- a/include/mesos/slave/isolator.proto
+++ b/include/mesos/slave/isolator.proto
@@ -111,7 +111,10 @@ message ContainerLaunchInfo {
optional Environment environment = 2;
// The root filesystem for the container.
- optional string rootfs = 3;
+ //
+ // This field is not used since 0.28, and it is deprecated
+ // since 1.0.1.
+ optional string rootfs = 3 [deprecated = true]; // Since 1.0.1.
// (Linux only) The namespaces required for the container.
// The namespaces are created while launching the executor.
[6/6] mesos git commit: Added MESOS-5927 to 1.0.1 CHANGELOG.
Posted by ji...@apache.org.
Added MESOS-5927 to 1.0.1 CHANGELOG.
Review: https://reviews.apache.org/r/50728/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/f0cf8cc3
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/f0cf8cc3
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/f0cf8cc3
Branch: refs/heads/master
Commit: f0cf8cc37484c1fc6c54f1fb0e260da5f54f1566
Parents: 1afdafd
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:59:50 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:59:50 2016 -0700
----------------------------------------------------------------------
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/f0cf8cc3/CHANGELOG
----------------------------------------------------------------------
diff --git a/CHANGELOG b/CHANGELOG
index eca695c..a395317 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,7 @@ All Issues:
* [MESOS-5911] - Webui redirection to leader in browser does not work.
* [MESOS-5913] - Stale socket FD usage when using libevent + SSL.
* [MESOS-5923] - Ubuntu 14.04 LTS GPU Isolator "/run" directory is noexec.
+ * [MESOS-5927] - Unable to run "scratch" Dockerfiles with Unified Containerizer.
* [MESOS-5943] - Incremental http parsing of URLs leads to decoder error.
* [MESOS-5945] - NvidiaVolume::create() should check for root before creating volume.
* [MESOS-5959] - All non-root tests fail on GPU machine.
[2/6] mesos git commit: Supported scratch containers in Unified
Containerizer.
Posted by ji...@apache.org.
Supported scratch containers in Unified Containerizer.
Before this patch, we are assuming the 'tmpfs' mount point '/tmp'
always exist in the container's new rootfs. However, this is not
true. For the scratch container (which is a common case), '/tmp'
may not exist, and there may only be an executable binary in the
new rootfs. So we need to create the mount point for 'tmpfs' in
fs::enter().
However, this change may break some cases using the bind backend,
because we are not able the create the '/tmp' mount point in a
read-only filesystem. So we require users to make sure the
directory '/tmp' must already exist in their single layer scratch
images for the following reasons:
1. For most cases, when operators prefer using the bind backend,
the single-layer images are usually large in size, and '/tmp'
exists in the rootfs.
2. For scratch images, most of them contain more than one layer,
which means the bind backend cannot be used in those cases.
So we can create the '/tmp' mount point if it does not exist.
3. If this is strictly a single layer scratch image, it is
reasonable that we require users to make sure the mount point
'/tmp' existed in the image rootfs if they are using the bind
backend, because we already require the sandbox mount point
to be existed in those iamges.
Review: https://reviews.apache.org/r/50719/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/98ffcb1b
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/98ffcb1b
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/98ffcb1b
Branch: refs/heads/master
Commit: 98ffcb1b69b4e233beb428bb63fc99d9e09245c2
Parents: 1cd3602
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:49:43 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:49:43 2016 -0700
----------------------------------------------------------------------
src/linux/fs.cpp | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/98ffcb1b/src/linux/fs.cpp
----------------------------------------------------------------------
diff --git a/src/linux/fs.cpp b/src/linux/fs.cpp
index 27d322c..f57db80 100644
--- a/src/linux/fs.cpp
+++ b/src/linux/fs.cpp
@@ -601,8 +601,15 @@ Try<Nothing> enter(const string& root)
// new root is writable (i.e., it could be a read only filesystem).
// Therefore, we always mount a tmpfs on /tmp in the new root so
// that we can create the mount point for the old root.
- if (!os::exists(path::join(root, "tmp"))) {
- return Error("/tmp in chroot does not exist");
+ //
+ // NOTE: If the new root is a read-only filesystem (e.g., using bind
+ // backend), the 'tmpfs' mount point '/tmp' must already exist in the
+ // new root. Otherwise, mkdir would return an error because of unable
+ // to create it in read-only filesystem.
+ Try<Nothing> mkdir = os::mkdir(path::join(root, "tmp"));
+ if (mkdir.isError()) {
+ return Error("Failed to create 'tmpfs' mount point at '" +
+ path::join(root, "tmp") + "': " + mkdir.error());
}
// TODO(jieyu): Consider limiting the size of the tmpfs.
[5/6] mesos git commit: Added a unit test for scratch based docker
images.
Posted by ji...@apache.org.
Added a unit test for scratch based docker images.
Review: https://reviews.apache.org/r/50727/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/1afdafd5
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/1afdafd5
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/1afdafd5
Branch: refs/heads/master
Commit: 1afdafd523389d73c4f8c1454bd95aebe051eabc
Parents: 0a35c42
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:50:07 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:50:07 2016 -0700
----------------------------------------------------------------------
.../containerizer/provisioner_docker_tests.cpp | 75 ++++++++++++++++++++
1 file changed, 75 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/1afdafd5/src/tests/containerizer/provisioner_docker_tests.cpp
----------------------------------------------------------------------
diff --git a/src/tests/containerizer/provisioner_docker_tests.cpp b/src/tests/containerizer/provisioner_docker_tests.cpp
index ffe3382..10fbc41 100644
--- a/src/tests/containerizer/provisioner_docker_tests.cpp
+++ b/src/tests/containerizer/provisioner_docker_tests.cpp
@@ -627,6 +627,81 @@ TEST_F(ProvisionerDockerPullerTest, ROOT_INTERNET_CURL_Whiteout)
driver.stop();
driver.join();
}
+
+
+// This test verifies that the scratch based docker image (that
+// only contain a single binary and its dependencies) can be
+// launched correctly.
+TEST_F(ProvisionerDockerPullerTest, ROOT_INTERNET_CURL_ScratchImage)
+{
+ Try<Owned<cluster::Master>> master = StartMaster();
+ ASSERT_SOME(master);
+
+ slave::Flags flags = CreateSlaveFlags();
+ flags.isolation = "docker/runtime,filesystem/linux";
+ flags.image_providers = "docker";
+
+ Owned<MasterDetector> detector = master.get()->createDetector();
+ Try<Owned<cluster::Slave>> slave = StartSlave(detector.get(), flags);
+ ASSERT_SOME(slave);
+
+ MockScheduler sched;
+ MesosSchedulerDriver driver(
+ &sched, DEFAULT_FRAMEWORK_INFO, master.get()->pid, DEFAULT_CREDENTIAL);
+
+ EXPECT_CALL(sched, registered(&driver, _, _));
+
+ Future<vector<Offer>> offers;
+ EXPECT_CALL(sched, resourceOffers(&driver, _))
+ .WillOnce(FutureArg<1>(&offers))
+ .WillRepeatedly(Return()); // Ignore subsequent offers.
+
+ driver.start();
+
+ AWAIT_READY(offers);
+ ASSERT_EQ(1u, offers->size());
+
+ const Offer& offer = offers.get()[0];
+
+ CommandInfo command;
+ command.set_shell(false);
+
+ TaskInfo task = createTask(
+ offer.slave_id(),
+ Resources::parse("cpus:1;mem:128").get(),
+ command);
+
+ Image image;
+ image.set_type(Image::DOCKER);
+
+ // 'hello-world' is a scratch image. It contains only one
+ // binary 'hello' in its rootfs.
+ image.mutable_docker()->set_name("hello-world");
+
+ ContainerInfo* container = task.mutable_container();
+ container->set_type(ContainerInfo::MESOS);
+ container->mutable_mesos()->mutable_image()->CopyFrom(image);
+
+ Future<TaskStatus> statusRunning;
+ Future<TaskStatus> statusFinished;
+ EXPECT_CALL(sched, statusUpdate(&driver, _))
+ .WillOnce(FutureArg<1>(&statusRunning))
+ .WillOnce(FutureArg<1>(&statusFinished));
+
+ driver.launchTasks(offer.id(), {task});
+
+ AWAIT_READY_FOR(statusRunning, Seconds(60));
+ EXPECT_EQ(task.task_id(), statusRunning->task_id());
+ EXPECT_EQ(TASK_RUNNING, statusRunning->state());
+
+ AWAIT_READY(statusFinished);
+ EXPECT_EQ(task.task_id(), statusFinished->task_id());
+ EXPECT_EQ(TASK_FINISHED, statusFinished->state());
+
+ driver.stop();
+ driver.join();
+}
+
#endif
} // namespace tests {
[4/6] mesos git commit: Fixed the bind backend document in
container-image.md.
Posted by ji...@apache.org.
Fixed the bind backend document in container-image.md.
Review: https://reviews.apache.org/r/50721/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0a35c429
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0a35c429
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/0a35c429
Branch: refs/heads/master
Commit: 0a35c429a3cf6752354dacf8ca70d036a15ad1ec
Parents: 5aa66ff
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:50:01 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:50:01 2016 -0700
----------------------------------------------------------------------
docs/container-image.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/0a35c429/docs/container-image.md
----------------------------------------------------------------------
diff --git a/docs/container-image.md b/docs/container-image.md
index 745f6b0..0feb028 100644
--- a/docs/container-image.md
+++ b/docs/container-image.md
@@ -311,10 +311,10 @@ share the source. Select writable areas can be achieved by mounting
read-write volumes to places like `/tmp`, `/var/tmp`, `/home`, etc.
using the `ContainerInfo`. These can be relative to the executor work
directory. Since the filesystem is read-only, `--sandbox_directory`
-must already exist within the filesystem because the filesystem
-isolator is unable to create it (e.g., either the image writer needs
-to create the mount point in the image, or the operator needs to set
-agent flag `--sandbox_directory` properly).
+and `/tmp` must already exist within the filesystem because the
+filesystem isolator is unable to create it (e.g., either the image
+writer needs to create the mount point in the image, or the operator
+needs to set agent flag `--sandbox_directory` properly).
### Overlay
[3/6] mesos git commit: Added comments for tmpfs and sandbox
mountpoints for bind backend.
Posted by ji...@apache.org.
Added comments for tmpfs and sandbox mountpoints for bind backend.
Review: https://reviews.apache.org/r/50720/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5aa66ff1
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5aa66ff1
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5aa66ff1
Branch: refs/heads/master
Commit: 5aa66ff187370168389628b8d447b4e83c3dbd22
Parents: 98ffcb1
Author: Gilbert Song <so...@gmail.com>
Authored: Tue Aug 2 16:49:58 2016 -0700
Committer: Jie Yu <yu...@gmail.com>
Committed: Tue Aug 2 16:49:58 2016 -0700
----------------------------------------------------------------------
.../containerizer/mesos/isolators/filesystem/linux.cpp | 3 +++
.../containerizer/mesos/provisioner/backends/bind.hpp | 10 +++++++---
2 files changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/5aa66ff1/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
----------------------------------------------------------------------
diff --git a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
index 31aa3e7..afcf978 100644
--- a/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
+++ b/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp
@@ -340,6 +340,9 @@ Try<vector<CommandInfo>> LinuxFilesystemIsolatorProcess::getPreExecCommands(
containerConfig.rootfs(),
flags.sandbox_directory);
+ // If the rootfs is a read-only filesystem (e.g., using the bind
+ // backend), the sandbox must be already exist. Please see the
+ // comments in 'provisioner/backend.hpp' for details.
Try<Nothing> mkdir = os::mkdir(sandbox);
if (mkdir.isError()) {
return Error(
http://git-wip-us.apache.org/repos/asf/mesos/blob/5aa66ff1/src/slave/containerizer/mesos/provisioner/backends/bind.hpp
----------------------------------------------------------------------
diff --git a/src/slave/containerizer/mesos/provisioner/backends/bind.hpp b/src/slave/containerizer/mesos/provisioner/backends/bind.hpp
index 2b2fcdc..19d44c6 100644
--- a/src/slave/containerizer/mesos/provisioner/backends/bind.hpp
+++ b/src/slave/containerizer/mesos/provisioner/backends/bind.hpp
@@ -39,9 +39,13 @@ class BindBackendProcess;
// mounting read-write volumes to places like /tmp, /var/tmp,
// /home, etc. using the ContainerInfo. These can be relative to
// the executor work directory.
-// N.B. Since the filesystem is read-only, '--sandbox_directory' must
-// already exist within the filesystem because the filesystem isolator
-// is unable to create it!
+// N.B. Since the filesystem is read-only:
+// i. The '--sandbox_directory' must already exist within the
+// filesystem because the filesystem isolator is unable to
+// create it!
+// ii. The 'tmpfs' moint point '/tmp' must already exist within
+// the filesystem, because 'pivot_root' needs a mount point
+// for the old root.
// 3) It's fast because the bind mount requires (nearly) zero IO.
class BindBackend : public Backend
{