You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Gilbert Song (JIRA)" <ji...@apache.org> on 2017/01/26 01:57:26 UTC
[jira] [Commented] (MESOS-6653) Overlayfs backend may fail to mount
the rootfs if both container image and image volume are specified.
[ https://issues.apache.org/jira/browse/MESOS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839028#comment-15839028 ]
Gilbert Song commented on MESOS-6653:
-------------------------------------
commit 27a5016154384eb18ec5d7577c1723c9d17e87f7
Author: Gilbert Song <so...@gmail.com>
Date: Wed Jan 25 07:51:35 2017 -0800
Fixed overlay backend provisioning multi images symlink.
Since the fix of MESOS-6000, symlinks are used in overlayfs
backend to shorten the arguments when mounting the rootfs.
E.g., '.../backends/overlay/links' is the symlink created
for a provisioned image. It becomes problematic if a
container image is specified while some image volumes are
specified for the same container. An unique symlink is
needed for each image to be provisioned.
Please note that changing the symlinks directory would
still be backward compatible for legacy containers, since
the container backend directory will be removed anyway in
provisioner::destroy().
Review: https://reviews.apache.org/r/54212/
> Overlayfs backend may fail to mount the rootfs if both container image and image volume are specified.
> ------------------------------------------------------------------------------------------------------
>
> Key: MESOS-6653
> URL: https://issues.apache.org/jira/browse/MESOS-6653
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Reporter: Gilbert Song
> Assignee: Gilbert Song
> Priority: Critical
> Labels: backend, containerizer, overlayfs
>
> Depending on MESOS-6000, we use symlink to shorten the overlayfs mounting arguments. However, if more than one image need to be provisioned (e.g., a container image is specified while image volumes are specified for the same container), the symlink .../backends/overlay/links would fail to be created since it exists already.
> Here is a simple log when we hard code overlayfs as our default backend:
> {noformat}
> [07:02:45] : [Step 10/10] [ RUN ] Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0
> [07:02:46] : [Step 10/10] I1127 07:02:46.416021 2919 containerizer.cpp:207] Using isolation: filesystem/linux,volume/image,docker/runtime,network/cni
> [07:02:46] : [Step 10/10] I1127 07:02:46.419312 2919 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> [07:02:46] : [Step 10/10] E1127 07:02:46.425336 2919 shell.hpp:107] Command 'hadoop version 2>&1' failed; this is the output:
> [07:02:46] : [Step 10/10] sh: 1: hadoop: not found
> [07:02:46] : [Step 10/10] I1127 07:02:46.425379 2919 fetcher.cpp:69] Skipping URI fetcher plugin 'hadoop' as it could not be created: Failed to create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was either not found or exited with a non-zero exit status: 127
> [07:02:46] : [Step 10/10] I1127 07:02:46.425452 2919 local_puller.cpp:94] Creating local puller with docker registry '/tmp/R6OUei/registry'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427258 2934 containerizer.cpp:956] Starting container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 for executor 'test_executor' of framework
> [07:02:46] : [Step 10/10] I1127 07:02:46.427592 2938 metadata_manager.cpp:167] Looking for image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427774 2936 local_puller.cpp:147] Untarring image 'test_image_rootfs' from '/tmp/R6OUei/registry/test_image_rootfs.tar' to '/tmp/R6OUei/store/staging/9krDz2'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512070 2933 local_puller.cpp:167] The repositories JSON file for image 'test_image_rootfs' is '{"test_image_rootfs":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512279 2933 local_puller.cpp:295] Extracting layer tar ball '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar to rootfs '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617442 2937 metadata_manager.cpp:155] Successfully cached image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617908 2938 provisioner.cpp:286] Image layers: 1
> [07:02:46] : [Step 10/10] I1127 07:02:46.617925 2938 provisioner.cpp:296] Should hit here
> [07:02:46] : [Step 10/10] I1127 07:02:46.617949 2938 provisioner.cpp:315] !!!!: bind
> [07:02:46] : [Step 10/10] I1127 07:02:46.617959 2938 provisioner.cpp:315] !!!!: overlay
> [07:02:46] : [Step 10/10] I1127 07:02:46.617967 2938 provisioner.cpp:315] !!!!: copy
> [07:02:46] : [Step 10/10] I1127 07:02:46.617974 2938 provisioner.cpp:318] Provisioning image rootfs '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7' for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] : [Step 10/10] I1127 07:02:46.618408 2936 overlay.cpp:175] Created symlink '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links' -> '/tmp/DQ3blT'
> [07:02:46] : [Step 10/10] I1127 07:02:46.618472 2936 overlay.cpp:203] Provisioning image rootfs with overlayfs: 'lowerdir=/tmp/DQ3blT/0,upperdir=/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/scratch/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7/upperdir,workdir=/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/scratch/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7/workdir'
> [07:02:46] : [Step 10/10] I1127 07:02:46.619098 2933 linux.cpp:451] Ignored an image volume for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330
> [07:02:46] : [Step 10/10] I1127 07:02:46.619745 2938 metadata_manager.cpp:167] Looking for image 'test_image_volume'
> [07:02:46] : [Step 10/10] I1127 07:02:46.619925 2937 local_puller.cpp:147] Untarring image 'test_image_volume' from '/tmp/R6OUei/registry/test_image_volume.tar' to '/tmp/R6OUei/store/staging/2GNlJO'
> [07:02:46] : [Step 10/10] I1127 07:02:46.713526 2935 local_puller.cpp:167] The repositories JSON file for image 'test_image_volume' is '{"test_image_volume":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] : [Step 10/10] I1127 07:02:46.713726 2935 local_puller.cpp:295] Extracting layer tar ball '/tmp/R6OUei/store/staging/2GNlJO/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar to rootfs '/tmp/R6OUei/store/staging/2GNlJO/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.818696 2937 metadata_manager.cpp:155] Successfully cached image 'test_image_volume'
> [07:02:46] : [Step 10/10] I1127 07:02:46.819169 2934 provisioner.cpp:286] Image layers: 1
> [07:02:46] : [Step 10/10] I1127 07:02:46.819188 2934 provisioner.cpp:296] Should hit here
> [07:02:46] : [Step 10/10] I1127 07:02:46.819221 2934 provisioner.cpp:315] !!!!: bind
> [07:02:46] : [Step 10/10] I1127 07:02:46.819232 2934 provisioner.cpp:315] !!!!: overlay
> [07:02:46] : [Step 10/10] I1127 07:02:46.819236 2934 provisioner.cpp:315] !!!!: copy
> [07:02:46] : [Step 10/10] I1127 07:02:46.819241 2934 provisioner.cpp:318] Provisioning image rootfs '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/baf632b3-29c5-45e4-9d2e-6f3a2bdd9759' for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] : [Step 10/10] ../../src/tests/containerizer/volume_image_isolator_tests.cpp:214: Failure
> [07:02:46] : [Step 10/10] (launch).failure(): Failed to create symlink '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links' -> '/tmp/6dj9IG'
> [07:02:46] : [Step 10/10] [ FAILED ] Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0, where GetParam() = false (919 ms)
> {noformat}
> We should differenciate the links for different provisioned images.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)