You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Avinash Sridharan (JIRA)" <ji...@apache.org> on 2016/10/13 02:42:20 UTC
[jira] [Commented] (MESOS-6143) resolv.conf is not copied when
using the Mesos containerizer with a Docker image
[ https://issues.apache.org/jira/browse/MESOS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570610#comment-15570610 ]
Avinash Sridharan commented on MESOS-6143:
------------------------------------------
[~jpinkul] tried Mesos 1.0.0 with an alpine image and I don't see this issue. I ran the Mesos 1.0.0 on Debian 8 with the following configuration:
Mesos master:
```
sudo /usr/sbin/mesos-master --ip=172.31.12.173 --port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
```
Mesos agent:
```
sudo /usr/sbin/mesos-slave --ip=172.31.12.173 --master=172.31.12.173:5050 --isolation=filesystem/linux,docker/runtime --work_dir=/var/lib/mesos --image_providers=docker
```
mesos-execute:
```
mesos-execute --master=172.31.12.173:5050 --name=dns-test --docker_image=alpine --command="sleep 10000"
```
Ran the `nsenter` command on the container and verified that the /etc/resolv.conf in the new mnt namespace is the same as that on the hostfs:
```
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ ps aux | grep mesos
admin 550 0.0 0.0 25540 2712 pts/0 S+ 01:52 0:00 screen -S mesos
admin 551 0.0 0.0 26900 3856 ? Ss 01:52 0:00 SCREEN -S mesos
root 1522 0.0 0.0 40540 3440 pts/1 S+ 01:59 0:00 sudo /usr/sbin/mesos-master --ip=172.31.12.173 --port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
root 1523 0.0 0.2 904584 33708 pts/1 Sl+ 01:59 0:01 /usr/sbin/mesos-master --ip=172.31.12.173 --port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
root 1538 0.0 0.0 40540 3428 pts/2 S+ 02:01 0:00 sudo /usr/sbin/mesos-slave --ip=172.31.12.173 --master=172.31.12.173:5050 --isolation=filesystem/linux,docker/runtime --work_dir=/var/lib/mesos --image_providers=docker
root 1539 0.1 0.2 818592 35980 pts/2 Sl+ 02:01 0:03 /usr/sbin/mesos-slave --ip=172.31.12.173 --master=172.31.12.173:5050 --isolation=filesystem/linux,docker/runtime --work_dir=/var/lib/mesos --image_providers=docker
admin 2045 0.0 0.1 817848 30744 pts/3 Sl+ 02:21 0:00 mesos-execute --master=172.31.12.173:5050 --name=dns-test --docker_image=alpine --command=sleep 10000
root 2058 0.2 0.1 816488 30068 ? Ssl 02:21 0:01 mesos-executor --launcher_dir=/usr/libexec/mesos --sandbox_directory=/mnt/mesos/sandbox --user=admin --rootfs=/var/lib/mesos/provisioner/containers/dcbe7b8a-e430-4b7f-98eb-d7f62c0c0f87/backends/copy/rootfses/0d8eceac-721b-4a3c-a68e-34ecc30cd718
admin 2101 0.0 0.0 12728 2168 pts/4 S+ 02:31 0:00 grep mesos
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ sudo nsenter -t 2058 -m cat /etc/alpine-release
3.4.3
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ ls /etc/
adduser.conf cloud deluser.conf grub.d initramfs-tools ld.so.cache lvm mke2fs.conf opt python2.7 rcS.d sgml subuid- udev
adjtime cron.d dhcp gshadow inputrc ld.so.conf machine-id modprobe.d os-release python3 resolv.conf shadow sudoers ufw
alternatives cron.daily dkms gshadow- insserv ld.so.conf.d magic modules pam.conf python3.4 rmt shadow- sudoers.d vim
apt cron.hourly dpkg gss insserv.conf libaudit.conf magic.mime modules-load.d pam.d rc0.d rpc shells sysconfig wgetrc
bash.bashrc cron.monthly emacs host.conf insserv.conf.d locale.alias mailcap motd passwd rc1.d rsyslog.conf skel sysctl.conf xdg
bash_completion.d crontab environment hostname iproute2 locale.gen mailcap.order mtab passwd- rc2.d rsyslog.d ssh sysctl.d xml
bindresvport.blacklist cron.weekly fstab hosts issue localtime manpath.config nanorc perl rc3.d screenrc ssl systemd
binfmt.d dbus-1 gai.conf hosts.allow issue.net logcheck mesos network profile rc4.d securetty staff-group-for-usr-local terminfo
ca-certificates debconf.conf groff hosts.deny java login.defs mesos-master networks profile.d rc5.d security subgid timezone
ca-certificates.conf debian_version group init kernel logrotate.conf mesos-slave newt protocols rc6.d selinux subgid- tmpfiles.d
calendar default group- init.d ldap logrotate.d mime.types nsswitch.conf python rc.local services subuid ucf.conf
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ ls /etc/apline-release
ls: cannot access /etc/apline-release: No such file or directory
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ sudo nsenter -t 2058 -m cat /etc/resolv.conf
domain us-west-2.compute.internal
search us-west-2.compute.internal
nameserver 172.31.0.2
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$ sudo cat /etc/resolv.conf
domain us-west-2.compute.internal
search us-west-2.compute.internal
nameserver 172.31.0.2
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
```
As you can see in the above output for PID 2058 , just to verify that we are looking at two different mount spaces listed the /etc of the hostfs and the /etc in the mnt namespace seen by PID 2058, and you can see the `alpine-release` file under /etc for PID 2058.
A few more questions on your test, given that you are saying the `resolv.conf` in your setup is empty in the container how is the `ping` that you are running even succeeding. If `resolv.conf` hadn't been setup properly, the container should have returned a TASK_FAILED?
> resolv.conf is not copied when using the Mesos containerizer with a Docker image
> --------------------------------------------------------------------------------
>
> Key: MESOS-6143
> URL: https://issues.apache.org/jira/browse/MESOS-6143
> Project: Mesos
> Issue Type: Bug
> Components: containerization, isolation
> Affects Versions: 1.0.0
> Environment: OS: Debian Jessie
> Mesos version: 1.0.0
> Reporter: Justin Pinkul
> Assignee: Avinash Sridharan
>
> When using the Mesos containierizer, host networking and a Docker image {{resolv.conf}} is not copied from the host. The only piece of Mesos code that copies these file is currently in the {{network/cni}} isolator so I tried turning this on, by setting {{isolation=network/cni,namespaces/pid,docker/runtime,cgroups/devices,gpu/nvidia,cgroups/cpu,disk/du,filesystem/linux}}, but the issue still remained. I suspect this might be related to not setting {{network_cni_config_dir}} and {{network_cni_plugins_dir}} but it seems incorrect that these flags would be required to use host networking.
> Here is how I am able to reproduce this issue:
> {code}
> mesos-execute --master=mesosmaster1:5050 \
> --name=dns-test \
> --docker_image=my-docker-image:1.1.3 \
> --command="bash -c 'ping google.com; while ((1)); do date; sleep 10; done'"
> # Find the PID of mesos-executor's child process and enter it
> nsenter -m -u -i -n -p -r -w -t $PID
> # This file will be empty
> cat /etc/resolv.conf
> {code}
> {code:title=Mesos agent log}
> I0908 17:39:24.599149 181564 slave.cpp:1688] Launching task dns-test for framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:24.599567 181564 paths.cpp:528] Trying to chown '/mnt/01/mesos_work/slaves/67025326-9dfd-4cbb-a008-454a40bce2f5-S2/frameworks/51831498-0902-4ae9-a1ff-4396f8b8d823-0006/executors/dns-test/runs/52bdce71-04b0-4440-bb71-cb826f0635c6' to user 'root'
> I0908 17:39:24.603970 181564 slave.cpp:5748] Launching executor dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/01/mesos_work/slaves/67025326-9dfd-4cbb-a008-454a40bce2f5-S2/frameworks/51831498-0902-4ae9-a1ff-4396f8b8d823-0006/executors/dns-test/runs/52bdce71-04b0-4440-bb71-cb826f0635c6'
> I0908 17:39:24.604178 181564 slave.cpp:1914] Queuing task 'dns-test' for executor 'dns-test' of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:24.604284 181571 docker.cpp:1020] Skipping non-docker container
> I0908 17:39:24.604532 181578 containerizer.cpp:781] Starting container '52bdce71-04b0-4440-bb71-cb826f0635c6' for executor 'dns-test' of framework '51831498-0902-4ae9-a1ff-4396f8b8d823-0006'
> I0908 17:39:24.606972 181571 provisioner.cpp:294] Provisioning image rootfs '/mnt/01/mesos_work/provisioner/containers/52bdce71-04b0-4440-bb71-cb826f0635c6/backends/copy/rootfses/db97ba50-c9f0-45e7-8a39-871e4038abf9' for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.037472 181564 cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1) for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.038415 181560 linux_launcher.cpp:281] Cloning child process with flags = CLONE_NEWNS | CLONE_NEWPID
> I0908 17:39:30.040742 181560 systemd.cpp:96] Assigned child process '190563' to 'mesos_executors.slice'
> I0908 17:39:30.161613 181576 slave.cpp:2902] Got registration for executor 'dns-test' of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 from executor(1)@10.191.4.65:43707
> I0908 17:39:30.162148 181563 disk.cpp:171] Updating the disk resources for container 52bdce71-04b0-4440-bb71-cb826f0635c6 to cpus(*):0.1; mem(*):32; gpus(*):2
> I0908 17:39:30.162648 181566 cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1) for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.162822 181574 slave.cpp:2079] Sending queued task 'dns-test' to executor 'dns-test' of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 at executor(1)@10.191.4.65:43707
> I0908 17:39:30.168383 181570 slave.cpp:3285] Handling status update TASK_RUNNING (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 from executor(1)@10.191.4.65:43707
> I0908 17:39:30.169019 181577 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:30.169173 181576 slave.cpp:3678] Forwarding the update TASK_RUNNING (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 to master@10.191.248.194:5050
> I0908 17:39:30.169242 181576 slave.cpp:3588] Sending acknowledgement for status update TASK_RUNNING (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 to executor(1)@10.191.4.65:43707
> I0908 17:39:30.171314 181565 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)