You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Qian Zhang (JIRA)" <ji...@apache.org> on 2018/02/08 13:50:00 UTC
[jira] [Comment Edited] (MESOS-8279) Persistent volumes are not
visible in Mesos UI using default executor on Linux.
[ https://issues.apache.org/jira/browse/MESOS-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356942#comment-16356942 ]
Qian Zhang edited comment on MESOS-8279 at 2/8/18 1:49 PM:
-----------------------------------------------------------
The above fix can handle this case: Framework launches a task group which has a task using a persistent volume disk, like below:
{code:java}
{
"tasks":[
{
"name": "test",
"task_id": {"value" : "test"},
"agent_id": {"value" : ""},
"resources": [
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 0.1
}
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 32
}
},
{
"name": "disk",
"type": "SCALAR",
"scalar": {
"value": 1024
},
"disk": {
"persistence": {
"id" : "pv1"
},
"volume": {
"mode": "RW",
"container_path": "xxx"
}
}
}
],
"command": {
"value": "echo hello > xxx/data && sleep 1000"
}
}
]
}
{code}
But it can not handle the case that framework launches a task group inside which the executor use a persistent volume disk and the task's {{containerInfo}} has a volume of {{SANDBOX_PATH}} type (like below).
{code:java}
"container": {
"type": "MESOS",
"volumes": [
{
"mode": "RW",
"container_path": "xxx",
"source": {
"type": "SANDBOX_PATH",
"sandbox_path": {
"type": "PARENT",
"path": "foo"
}
}
}
]
}
{code}
I think this is how Marathon launches a pod (task group). If you use Marathon to launch a pod and specify a PV for the container inside of the pod, actually the executor will have a disk resource which uses that PV and Marathon will create the PV and mount it into the executor's sandbox, and the task info created by Marathon will have a volume of {{SANDBOX_PATH}} type like the above so the task can share the PV with the executor. In this case, the fix to this ticket will not do the file attach since it can only handle the case of task using persistent volume disk resource, see [this code|https://github.com/apache/mesos/blob/32b85a2b06f676b68a16deaa8359ae64a1e8ead9/src/slave/slave.cpp#L1035:L1039] for details.
To fix this issue, I think we need to improve {{Slave::attachTaskVolumeDirectory}} to handle the task whose {{containerInfo}} has a volume of {{SANDBOX_PATH}} type, for such task, call {{Files::attach()}} to attach the executor's volume path to the task's volume path.
was (Author: qianzhang):
The above fix can handle this case: Framework launches a task group which has a task using a persistent volume disk, like below:
{code:java}
{
"tasks":[
{
"name": "test",
"task_id": {"value" : "test"},
"agent_id": {"value" : ""},
"resources": [
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 0.1
}
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 32
}
},
{
"name": "disk",
"type": "SCALAR",
"scalar": {
"value": 1024
},
"disk": {
"persistence": {
"id" : "pv1"
},
"volume": {
"mode": "RW",
"container_path": "xxx"
}
}
}
],
"command": {
"value": "echo hello > xxx/data && sleep 1000"
}
}
]
}
{code}
But it can not handle the case that framework launches a task group inside which the executor use a persistent volume disk and the task's {{containerInfo}} has a volume of {{SANDBOX_PATH}} type (like below).
{code:java}
"container": {
"type": "MESOS",
"volumes": [
{
"mode": "RW",
"container_path": "xxx",
"source": {
"type": "SANDBOX_PATH",
"sandbox_path": {
"type": "PARENT",
"path": "foo"
}
}
}
]
}
{code}
I think this is how Marathon launches a pod (task group). If you use Marathon to launch a pod and specify a PV for the container inside of the pod, the executor actually will have a disk resource which has that PV and Marathon will create the PV and mount it into the executor's sandbox, and the task info created by Marathon will have a volume of {{SANDBOX_PATH}} type like the above so that task can share the PV with the executor. In this case, the fix to this ticket will not do the file attach since it can only handle the case of task using persistent volume disk resource, see [this code|https://github.com/apache/mesos/blob/32b85a2b06f676b68a16deaa8359ae64a1e8ead9/src/slave/slave.cpp#L1035:L1039] for details.
To fix this issue, I was thinking to improve {{Slave::attachTaskVolumeDirectory}} to handle the task whose {{containerInfo}} has a volume of {{SANDBOX_PATH}} type: For such task, call {{Files::attach()}} to attach the executor's volume path to the task's volume path, but the problem is, the file attach will fail because the executor's volume path has not been created yet at this moment (sending task to executor), instead it will be created by the {{volume/sandbox_path}} isolator when the nested container corresponding to the task is launched.
> Persistent volumes are not visible in Mesos UI using default executor on Linux.
> -------------------------------------------------------------------------------
>
> Key: MESOS-8279
> URL: https://issues.apache.org/jira/browse/MESOS-8279
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.2.2, 1.3.1, 1.4.1
> Reporter: Jie Yu
> Assignee: Qian Zhang
> Priority: Major
> Fix For: 1.5.0, 1.6.0
>
>
> The reason is because on Linux, if multiple containers in a default executor want to share a persistent volume, it'll use SANDBOX_PATH volume source with type PARENT. This will be translated into a bind mount in the nested container's mount namespace, thus not visible in the host mount namespace. Mesos UI operates in the host mount namespace.
> One potential solution for that is to create a symlink (instead of just a mkdir) in the sandbox. The symlink will be shadowed by the bind mount in the nested container, but in the host mount namespace, it'll points to the corresponding persistent volume.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)