You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by qi...@apache.org on 2020/10/13 02:16:11 UTC
[mesos] branch master updated (c28fd3a -> 90e5434)
This is an automated email from the ASF dual-hosted git repository.
qianzhang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from c28fd3a Re-added the obsolete `updateFramework` signature into libmesos-java.so.
new 3e1e0b3 Added doc for the `volume/csi` isolator.
new 90e5434 Corrected the example of the managed CSI plugin.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/configuration/agent.md | 8 +-
docs/isolators/csi-volume.md | 339 +++++++++++++++++++++++++++++++++++++++++++
docs/mesos-containerizer.md | 1 +
src/slave/flags.cpp | 8 +-
4 files changed, 342 insertions(+), 14 deletions(-)
create mode 100644 docs/isolators/csi-volume.md
[mesos] 02/02: Corrected the example of the managed CSI plugin.
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qianzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 90e5434544da9886cd6f2d87b73e3246292af107
Author: Qian Zhang <zh...@gmail.com>
AuthorDate: Tue Oct 13 09:58:44 2020 +0800
Corrected the example of the managed CSI plugin.
Review: https://reviews.apache.org/r/72846
---
docs/configuration/agent.md | 8 +-------
src/slave/flags.cpp | 8 +-------
2 files changed, 2 insertions(+), 14 deletions(-)
diff --git a/docs/configuration/agent.md b/docs/configuration/agent.md
index 4899202..a8286d1 100644
--- a/docs/configuration/agent.md
+++ b/docs/configuration/agent.md
@@ -1528,16 +1528,10 @@ Example config files in this directory:
"containers": [
{
"services": [
- "CONTROLLER_SERVICE",
"NODE_SERVICE"
],
"command": {
- "shell": false,
- "value": "managed-plugin",
- "arguments": [
- "managed-plugin",
- "--endpoint=$(CSI_ENDPOINT)"
- ]
+ "value": "<path-to-managed-plugin> --endpoint=$CSI_ENDPOINT"
},
"resources": [
{"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
diff --git a/src/slave/flags.cpp b/src/slave/flags.cpp
index 878788c..e66b464 100644
--- a/src/slave/flags.cpp
+++ b/src/slave/flags.cpp
@@ -129,16 +129,10 @@ mesos::internal::slave::Flags::Flags()
" \"containers\": [\n"
" {\n"
" \"services\": [\n"
- " \"CONTROLLER_SERVICE\",\n"
" \"NODE_SERVICE\"\n"
" ],\n"
" \"command\": {\n"
- " \"shell\": false,\n"
- " \"value\": \"managed-plugin\",\n"
- " \"arguments\": [\n"
- " \"managed-plugin\",\n"
- " \"--endpoint=$(CSI_ENDPOINT)\"\n"
- " ]\n"
+ " \"value\": \"<path-to-managed-plugin> --endpoint=$CSI_ENDPOINT\"\n"
" },\n"
" \"resources\": [\n"
" {\"name\": \"cpus\", \"type\": \"SCALAR\", \"scalar\": {\"value\": 0.1}},\n" // NOLINT(whitespace/line_length)
[mesos] 01/02: Added doc for the `volume/csi` isolator.
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qianzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 3e1e0b37d6a30a2c98d1227b4ac754b1d26686f3
Author: Qian Zhang <zh...@gmail.com>
AuthorDate: Wed Sep 9 10:26:52 2020 +0800
Added doc for the `volume/csi` isolator.
Review: https://reviews.apache.org/r/72845
---
docs/isolators/csi-volume.md | 339 +++++++++++++++++++++++++++++++++++++++++++
docs/mesos-containerizer.md | 1 +
2 files changed, 340 insertions(+)
diff --git a/docs/isolators/csi-volume.md b/docs/isolators/csi-volume.md
new file mode 100644
index 0000000..1b1a635
--- /dev/null
+++ b/docs/isolators/csi-volume.md
@@ -0,0 +1,339 @@
+---
+title: Apache Mesos - CSI Volume Support in Mesos Containerizer
+layout: documentation
+---
+
+# Pre-provisioned CSI Volume Support in Mesos Containerizer
+
+Mesos 1.11.0 adds pre-provisioned CSI volume support to the
+[MesosContainerizer](../mesos-containerizer.md) (a.k.a., the universal
+containerizer) by introducing the new `volume/csi` isolator.
+
+This document describes the motivation and the configuration steps for enabling
+the `volume/csi` isolator, and required framework changes.
+
+## Table of Contents
+- [Motivation](#motivation)
+- [How does it work?](#how-does-it-work)
+- [Configuration](#configuration)
+ - [Pre-conditions](#pre-conditions)
+ - [Configuring CSI Volume Isolator](#configure-csi-volume-isolator)
+ - [Enabling frameworks to use CSI volumes](#enable-frameworks)
+ - [Volume Protobuf](#volume-protobuf)
+ - [Example](#example)
+
+## <a name="motivation"></a>Motivation
+
+[Container Storage Interface](https://github.com/container-storage-interface/spec)
+(CSI) is a specification that defines a common set of APIs for all interactions
+between the storage vendors and the container orchestration platforms. Building
+CSI support allows Mesos to make use of the quickly-growing CSI ecosystem.
+
+We already have a [solution](../csi.md) to support CSI introduced in the Mesos
+1.5.0 release, but that solution has a limitation: it requires CSI plugins to
+implement the [ListVolumes](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#listvolumes)
+and [GetCapacity](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#getcapacity)
+APIs so that the external storage can be modeled as Mesos raw disk resources and
+then offered to frameworks. However there are a lot of 3rd party CSI plugins the
+do not implement those two APIs.
+
+Mesos 1.11.0 provides a more generic way to support 3rd party CSI plugins so
+that Mesos can work with broader external storage ecosystem and we will benefit
+from continued development of the community CSI plugins.
+
+## <a name="how-does-it-work"></a>How does it work?
+
+The `volume/csi` isolator interacts with CSI plugins via the plugin's gRPC
+endpoint.
+
+When a new task with CSI volumes is launched, the `volume/csi` isolator will
+call the CSI plugin to publish the specified CSI volumes onto the agent host
+and then mount them onto the task container. When the task terminates, the
+`volume/csi` isolator will call the CSI plugin to unpublish the specified CSI
+volumes.
+
+Currently the `volume/csi` isolator will only call the CSI plugin's [node service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#node-service-rpc) but not [controller service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controller-service-rpc), that means:
+
+- We only support pre-provisioned CSI volume but not dynamic CSI volumes
+ provisioning, so operators need to create the CSI volumes explicitly and
+ provide the volume info (e.g. volume ID, context, etc.) to frameworks so
+ that frameworks can use the volumes in their tasks.
+
+- We do not support the CSI volumes that require the controller service to
+ publish to a node ([ControllerPublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controllerpublishvolume)) prior to the node service publishing on the node
+ ([NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume)).
+
+## <a name="configuration"></a>Configuration
+
+To use the `volume/csi` isolator, there are certain actions required by
+operators and framework developers. In this section we list the steps
+required by the operator to configure the `volume/csi` isolator and the steps
+required by framework developers to specify CSI volumes in their tasks.
+
+### <a name="pre-conditions"></a>Pre-conditions
+
+- Explicitly create the CSI volumes that are going to be accessed by Mesos
+ tasks. For some CSI plugins (e.g. [NFS](https://github.com/kubernetes-csi/csi-driver-nfs)),
+ they do not implement the [CreateVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#createvolume)
+ API, so operators do not need to create the volume explicitly in this case.
+
+### <a name="configure-csi-volume-isolator"></a>Configuring the CSI Volume Isolator
+
+In order to configure the `volume/csi` isolator, the operator needs to
+configure the `--isolation` and `--csi_plugin_config_dir` flags at agent
+startup as follows:
+
+```{.console}
+ sudo mesos-agent \
+ --master=<master-IP:master-port> \
+ --work_dir=/var/lib/mesos \
+ --isolation=filesystem/linux,volume/csi \
+ --csi_plugin_config_dir=<directory that contains CSI plugin configuration files>
+```
+
+The `volume/csi` isolator must be specified in the `--isolation` flag at agent
+startup; the `volume/csi` isolator has a dependency on the `filesystem/linux`
+isolator.
+
+The operator needs to put the CSI plugin configuration files under the directory
+specified via the agent flag `--csi_plugin_config_dir`. Each file in this
+directory should contain a JSON object representing a `CSIPluginInfo` object
+which can be either a managed CSI plugin (i.e. the plugin launched by Mesos as
+a standalone container) or an unmanaged CSI plugin (i.e. the plugin launched
+outside of Mesos).
+
+```{.proto}
+message CSIPluginInfo {
+ required string type = 1;
+ optional string name = 2 [default = "default"];
+
+ // A list of container configurations to run managed CSI plugin.
+ repeated CSIPluginContainerInfo containers = 3;
+
+ // The service endpoints of the unmanaged CSI plugin.
+ repeated CSIPluginEndpoint endpoints = 4;
+
+ optional string target_path_root = 5;
+ optional bool target_path_exists = 6;
+}
+
+message CSIPluginContainerInfo {
+ enum Service {
+ UNKNOWN = 0;
+ CONTROLLER_SERVICE = 1;
+ NODE_SERVICE = 2;
+ }
+
+ repeated Service services = 1;
+ optional CommandInfo command = 2;
+ repeated Resource resources = 3;
+ optional ContainerInfo container = 4;
+}
+
+message CSIPluginEndpoint {
+ required CSIPluginContainerInfo.Service csi_service = 1;
+ required string endpoint = 2;
+}
+```
+
+Example of managed CSI plugin:
+```{.json}
+{
+ "type": "org.apache.mesos.csi.managed-plugin",
+ "containers": [
+ {
+ "services": [
+ "NODE_SERVICE"
+ ],
+ "command": {
+ "value": "<path-to-managed-plugin> --endpoint=$CSI_ENDPOINT"
+ },
+ "resources": [
+ {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
+ {"name": "mem", "type": "SCALAR", "scalar": {"value": 1024}}
+ ]
+ }
+ ]
+}
+```
+
+Example of unmanaged CSI plugin:
+```{.json}
+{
+ "type": "org.apache.mesos.csi.unmanaged-plugin",
+ "endpoints": [
+ {
+ "csi_service": "NODE_SERVICE",
+ "endpoint": "/var/lib/unmanaged-plugin/csi.sock"
+ }
+ ],
+ "target_path_root": "/mnt/unmanaged-plugin"
+}
+```
+
+### <a name="enable-frameworks"></a>Enabling frameworks to use CSI volumes
+
+#### <a name="volume-protobuf"></a>Volume Protobuf
+
+The `Volume` protobuf message has been updated to support CSI volumes.
+
+```{.proto}
+message Volume {
+ ...
+ required Mode mode = 3;
+ required string container_path = 1;
+
+ message Source {
+ enum Type {
+ UNKNOWN = 0;
+ ...
+ CSI_VOLUME = 5;
+ }
+
+ message CSIVolume {
+ required string plugin_name = 1;
+
+ message VolumeCapability {
+ message BlockVolume {
+ }
+
+ message MountVolume {
+ optional string fs_type = 1;
+ repeated string mount_flags = 2;
+ }
+
+ message AccessMode {
+ enum Mode {
+ UNKNOWN = 0;
+ SINGLE_NODE_WRITER = 1;
+ SINGLE_NODE_READER_ONLY = 2;
+ MULTI_NODE_READER_ONLY = 3;
+ MULTI_NODE_SINGLE_WRITER = 4;
+ MULTI_NODE_MULTI_WRITER = 5;
+ }
+
+ required Mode mode = 1;
+ }
+
+ oneof access_type {
+ BlockVolume block = 1;
+ MountVolume mount = 2;
+ }
+
+ required AccessMode access_mode = 3;
+ }
+
+ // Specifies the parameters used to stage/publish a pre-provisioned volume
+ // on an agent host.
+ message StaticProvisioning {
+ required string volume_id = 1;
+ required VolumeCapability volume_capability = 2;
+ optional bool readonly = 3;
+ map<string, Secret> node_stage_secrets = 4;
+ map<string, Secret> node_publish_secrets = 5;
+ map<string, string> volume_context = 6;
+ }
+
+ optional StaticProvisioning static_provisioning = 2;
+ }
+
+ optional Type type = 1;
+ ...
+ optional CSIVolume csi_volume = 6;
+ }
+
+ optional Source source = 5;
+}
+```
+
+When requesting a CSI volume for a container, the framework developer needs to
+set `Volume` for the container, which includes `mode`, `container_path` and
+`source` fields.
+
+The `source` field specifies where the volume comes from. Framework developers
+need to set the `type` field to `CSI_VOLUME` and specify the `csi_volume` field.
+
+The `csi_volume` field specifies the information of the CSI volume. Framework
+developers need to set the `plugin_name` field to the `type` field of one of the
+CSI plugin configuration files in the directory specified via the agent flag
+`--csi_plugin_config_dir`, and specify the `static_provisioning` field according
+to the information of the pre-provisioned volume. The fields in `static_provisioning`
+map directly onto the fields in the CSI calls [NodeStageVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodestagevolume)
+and [NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume),
+please find more detailed descriptions about those fields in the CSI spec.
+
+How to specify `container_path`:
+
+1. If you are launching a task without a container image and `container_path`
+ is an absolute path, you need to make sure the absolute path exists on your
+ host root file system as the container shares the host root file system;
+ otherwise, the task will fail.
+
+2. For other cases like launching a task without a container image and with a
+ relative `container_path`, or launching a task with a container image and an
+ absolute or relative `container_path`, the `volume/csi` isolator will help
+ create the `container_path` as the mount point.
+
+The following table summarizes the above rules for `container_path`:
+
+<table class="table table-striped">
+ <tr>
+ <th></th>
+ <th>Task with rootfs</th>
+ <th>Task without rootfs</th>
+ </tr>
+ <tr>
+ <td>Absolute container_path</td>
+ <td>No need to exist</td>
+ <td>Must exist</td>
+ </tr>
+ <tr>
+ <td>Relative container_path</td>
+ <td>No need to exist</td>
+ <td>No need to exist</td>
+ </tr>
+</table>
+
+#### <a name="example"></a>Example
+
+Launch a task with a CSI volume managed by NFS CSI plugin:
+
+ ```{.json}
+ TaskInfo {
+ ...
+ "command" : {
+ "value": "echo test > volume/file"
+ },
+ "container" : {
+ "type": "MESOS",
+ "volumes" : [
+ {
+ "container_path" : "volume",
+ "mode" : "RW",
+ "source": {
+ "type": "CSI_VOLUME",
+ "csi_volume": {
+ "plugin_name": "nfs.csi.k8s.io",
+ "static_provisioning": {
+ "volume_id": "foo",
+ "volume_capability": {
+ "mount": {},
+ "access_mode": {
+ "mode": "MULTI_NODE_MULTI_WRITER"
+ }
+ },
+ "volume_context": {
+ "server": "192.168.1.100",
+ "share": "/mnt/data"
+ }
+ }
+ }
+ }
+ }
+ ]
+ }
+ }
+ ```
+
+**NOTE**: To make the above example work, an NFS server (`192.168.1.100`) needs to
+be setup to export the directory `/mnt/data`.
diff --git a/docs/mesos-containerizer.md b/docs/mesos-containerizer.md
index 3231cb9..c632550 100644
--- a/docs/mesos-containerizer.md
+++ b/docs/mesos-containerizer.md
@@ -59,6 +59,7 @@ Mesos supports the following built-in isolators.
- posix/cpu
- posix/mem
- [posix/rlimits](isolators/posix-rlimits.md)
+- [volume/csi](isolators/csi-volume.md)
- [volume/host_path](container-volume.md#host_path-volume-source)
- volume/image
- [volume/sandbox_path](container-volume.md#sandbox_path-volume-source)