Problems with tasks and cni networking after upgrading from 1.8 to 1.9

[Marc Roos <M....@f1-outsourcing.eu>]

Looks like my tasks that have dual networking, a gateway and cni_args 
assigned ip address are not able to start anymore on mesos 1.9. During 
deployment I am able to ping these assigned ip addresses. Why can't this 
executor reach the task then? I guess something has changed in how the 
executor is connecting with container since 1.8?



I0929 00:54:55.658519 469057 slave.cpp:2130] Got assigned task 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' for framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0929 00:54:55.664753 469057 slave.cpp:2504] Authorizing task 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' for framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0929 00:54:55.667817 469057 slave.cpp:2977] Launching task 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' for framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0929 00:54:55.668617 469057 paths.cpp:817] Creating sandbox 
'/var/lib/mesos/slaves/0c15c45f-310b-4fc2-8275-b0bfa1bdfdcb-S0/framework
s/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/demo_server-storag
e-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a1._app.3/runs/51e76
e90-5244-4c67-98bf-cf30b78d6fe6' for user 'nobody'
I0929 00:54:55.669667 469057 paths.cpp:820] Creating sandbox 
'/var/lib/mesos/meta/slaves/0c15c45f-310b-4fc2-8275-b0bfa1bdfdcb-S0/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/demo_server-s
torage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a1._app.3/runs/
51e76e90-5244-4c67-98bf-cf30b78d6fe6'
I0929 00:54:55.669903 469057 slave.cpp:10002] Launching executor 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' of framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 with 
resources 
[{"allocation_info":{"role":"marathon"},"name":"cpus","scalar":{"value":
0.1},"type":"SCALAR"},{"allocation_info":{"role":"marathon"},"name":"mem
","scalar":{"value":32.0},"type":"SCALAR"}] in work directory 
'/var/lib/mesos/slaves/0c15c45f-310b-4fc2-8275-b0bfa1bdfdcb-S0/framework
s/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/demo_server-storag
e-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a1._app.3/runs/51e76
e90-5244-4c67-98bf-cf30b78d6fe6'
I0929 00:54:55.670975 469057 slave.cpp:3209] Queued task 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' for executor 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' of framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0929 00:54:55.671329 469057 slave.cpp:3657] Launching container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6 for executor 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' of framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0929 00:54:55.671910 469042 containerizer.cpp:1396] Starting container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6
I0929 00:54:55.672205 469042 containerizer.cpp:3323] Transitioning the 
state of container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 from STARTING to 
PROVISIONING after 162048ns
I0929 00:54:55.973661 469039 provisioner.cpp:551] Provisioning image 
rootfs 
'/var/lib/mesos/provisioner/containers/51e76e90-5244-4c67-98bf-cf30b78d6
fe6/backends/copy/rootfses/292ee89d-1112-4b79-83e7-953571b3cd3b' for 
container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 using copy backend
I0929 00:54:57.279662 469057 containerizer.cpp:3323] Transitioning the 
state of container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 from 
PROVISIONING to PREPARING after 1.607405056secs
I0929 00:54:57.285080 469064 cpu.cpp:92] Updated 'cpu.shares' to 204 
(cpus 0.2) for container 51e76e90-5244-4c67-98bf-cf30b78d6fe6
I0929 00:54:57.296098 469057 switchboard.cpp:316] Container logger 
module finished preparing container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6; IOSwitchboard server is not 
required
I0929 00:54:57.298223 469064 linux_launcher.cpp:492] Launching container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6 and cloning with namespaces 
CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
I0929 00:54:57.307914 469057 containerizer.cpp:2209] Checkpointing 
container's forked pid 469525 to 
'/var/lib/mesos/meta/slaves/0c15c45f-310b-4fc2-8275-b0bfa1bdfdcb-S0/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/demo_server-s
torage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a1._app.3/runs/
51e76e90-5244-4c67-98bf-cf30b78d6fe6/pids/forked.pid'
I0929 00:54:57.308645 469057 containerizer.cpp:3323] Transitioning the 
state of container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 from PREPARING 
to ISOLATING after 29.022976ms
I0929 00:54:57.310788 469040 cni.cpp:974] Bind mounted 
'/proc/469525/ns/net' to 
'/run/mesos/isolators/network/cni/51e76e90-5244-4c67-98bf-cf30b78d6fe6/n
s' for container 51e76e90-5244-4c67-98bf-cf30b78d6fe6
I0929 00:54:57.311151 469040 cni.cpp:1320] Invoking CNI plugin 
'/usr/libexec/cni/mesos' to attach container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6 to network 'cni-apps-gw'
I0929 00:54:57.318881 469040 cni.cpp:1320] Invoking CNI plugin 
'/usr/libexec/cni/mesos' to attach container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6 to network 'cni-storage'
I0929 00:54:57.378106 469037 cni.cpp:1410] Got assigned IPv4 address 
'192.168.122.200/24' from CNI network 'cni-apps-gw' for container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6
I0929 00:54:57.378398 469037 cni.cpp:1410] Got assigned IPv4 address 
'192.168.10.200/24' from CNI network 'cni-storage' for container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6
I0929 00:54:57.378682 469037 cni.cpp:1117] DNS nameservers for container 
51e76e90-5244-4c67-98bf-cf30b78d6fe6 are: 212.26.193.10, 212.19.193.130, 
192.168.10.10, 192.168.10.12
I0929 00:54:57.579591 469039 containerizer.cpp:3323] Transitioning the 
state of container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 from ISOLATING 
to FETCHING after 270.804992ms
I0929 00:54:57.580142 469039 containerizer.cpp:3323] Transitioning the 
state of container 51e76e90-5244-4c67-98bf-cf30b78d6fe6 from FETCHING to 
RUNNING after 701952ns

I0929 00:55:55.671924 469042 slave.cpp:7354] Terminating executor 
'demo_server-storage-appsgw.instance-7239a528-e242-11e9-ac3c-0050563001a
1._app.3' of framework d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 beca
use it did not register within 1mins




- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. 
F1 Outsourcing Development Sp. z o.o.
Poland 

t:  +48 (0)124466845
f:  +48 (0)124466843
e:  marc@f1-outsourcing.eu