You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Olivier Sallou <ol...@irisa.fr> on 2016/08/23 16:05:07 UTC
Fail to get CNI with unified containerizer, job remains stuck on
staging
HI,
I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
Weave works nicely with the Docker containerizer.
When I try to launch a task via my framework with unified containerizer,
the job remains waiting forever (no RUNNING message). I can see however
that weave cni allocated an IP address to Mesos.
I tried with a simple mesos-execute test.
Example with a mesos-execute with no CNI, everything is OK
osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
--command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
--name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
master@127.0.0.1:5050
Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
Received status update TASK_RUNNING for task 'test0'
source: SOURCE_EXECUTOR
Received status update TASK_FINISHED for task 'test0'
message: 'Command exited with status 0'
Sample example specifying the weave network
osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
--command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
--name=test0 --networks=weave
I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
master@127.0.0.1:5050
Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
mesos-slave logs:
I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
'/proc/28869/ns/net' to
'/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
'10.32.0.1/12' from CNI network 'weave' for container
4f91a5df-2e9a-4cfc-93f5-aa197646db09
I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
'/etc/resolv.conf'
There are no other logs until I kill the job.
We can see that Mesos container got an IP but it seems to block on DNS,
Thanks for hints....
--
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to get CNI with unified containerizer, job remains stuck on
staging
Posted by Olivier Sallou <ol...@irisa.fr>.
On 08/24/2016 04:04 PM, Avinash Sridharan wrote:
> Oliver, you can't have the agent running on 127.0.0.1. The agent needs to
> be running in a routeabl IP address (choose an IP from one of the
> interfaces).
>
> Reason being that if agent is on local host the executor running in its own
> network namespace will try to make a connection in its own network
> namespace and fail.
Thanks! modying ip address to reachable IP works.....
> On Wed, Aug 24, 2016 at 5:15 AM Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>> I have the same behavior with Calico. Task get IP from CNI plugin, but
>> task remains in STAGING and same logs.
>>
>> mesos-execute --containerizer=mesos \
>>> --name=cni \
>>> --master=127.0.0.1:5050 \
>>> --networks=calico-net-1 \
>>> --command="ifconfig"
>> I0824 14:12:03.202328 24912 scheduler.cpp:172] Version: 1.0.0
>> I0824 14:12:03.203009 24911 scheduler.cpp:461] New master detected at
>> master@127.0.0.1:5050
>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0017'
>> Submitted task 'cni' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>
>> REMAINS STAGING!
>>
>>
>> I0824 14:12:03.857158 24806 cni.cpp:1109] Got assigned IPv4 address
>> '192.168.0.0/32' from CNI network 'calico-net-1' for container
>> bdbb275a-ec5f-4a50-aca0-5e694ae57324
>> I0824 14:12:03.857348 24805 cni.cpp:838] Unable to find DNS nameservers
>> for container bdbb275a-ec5f-4a50-aca0-5e694ae57324. Using host
>> '/etc/resolv.conf'
>>
>> No more logs
>>
>>
>> Olivier
>>
>> On 08/24/2016 08:23 AM, Olivier Sallou wrote:
>>> On 08/23/2016 06:13 PM, Jie Yu wrote:
>>>> The DNS related logging means that the weave plugin does not return DNS
>>>> information, the agent uses the host resolv.conf for the container. So I
>>>> think is irrelevant to your problem.
>>>>
>>>> Mesos requires that executor can talk to agent. Can you see if there is
>> a
>>>> route from 10.32.0.1 to the agent IP?
>>> How can I check this as task does not start ? I have exposed weave
>>> network on host:
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
>>> 10.32.0.2
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
>>> PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
>>> 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
>>> 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
>>> 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
>>> 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
>>>
>>> And why is it blocking?
>>>
>>> I am on a single host environement, so agent is on 127.0.0.1.
>>>
>>> Olivier
>>>> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <
>> olivier.sallou@irisa.fr>
>>>> wrote:
>>>>
>>>>> HI,
>>>>>
>>>>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>>>>
>>>>> Weave works nicely with the Docker containerizer.
>>>>>
>>>>> When I try to launch a task via my framework with unified
>> containerizer,
>>>>> the job remains waiting forever (no RUNNING message). I can see however
>>>>> that weave cni allocated an IP address to Mesos.
>>>>>
>>>>> I tried with a simple mesos-execute test.
>>>>>
>>>>> Example with a mesos-execute with no CNI, everything is OK
>>>>>
>>>>>
>>>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
>> mesos-execute
>>>>> --command="sleep 2" -docker_image=centos:latest --master=
>> 127.0.0.1:5050
>>>>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version:
>> 1.0.0
>>>>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>>>>> master@127.0.0.1:5050
>>>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>>>>> Submitted task 'test0' to agent
>> 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>>>> Received status update TASK_RUNNING for task 'test0'
>>>>> source: SOURCE_EXECUTOR
>>>>> Received status update TASK_FINISHED for task 'test0'
>>>>> message: 'Command exited with status 0'
>>>>>
>>>>>
>>>>> Sample example specifying the weave network
>>>>>
>>>>>
>>>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
>> mesos-execute
>>>>> --command="sleep 2" -docker_image=centos:latest --master=
>> 127.0.0.1:5050
>>>>> --name=test0 --networks=weave
>>>>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>>>>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>>>>> master@127.0.0.1:5050
>>>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>>>>> Submitted task 'test0' to agent
>> 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>>>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>>>>
>>>>> mesos-slave logs:
>>>>>
>>>>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>>>>> '/proc/28869/ns/net' to
>>>>>
>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>>>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>>>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>>>>> '10.32.0.1/12' from CNI network 'weave' for container
>>>>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>>>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>>>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>>>>> '/etc/resolv.conf'
>>>>>
>>>>> There are no other logs until I kill the job.
>>>>> We can see that Mesos container got an IP but it seems to block on DNS,
>>>>>
>>>>> Thanks for hints....
>>>>>
>>>>> --
>>>>>
>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>>>
>>>>>
>>>>>
>> --
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to get CNI with unified containerizer, job remains stuck on staging
Posted by Qian Zhang <zh...@gmail.com>.
Yes, the executor will try to connect to agent via 127.0.0.1:5051 in its
own network namespace, but your agent is actually listening on
127.0.0.1:5051 in the host network namespace, so executor can never connect
to agent. You need to have your agent listen on a host interface.
Thanks,
Qian Zhang
On Wed, Aug 24, 2016 at 10:04 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> Oliver, you can't have the agent running on 127.0.0.1. The agent needs to
> be running in a routeabl IP address (choose an IP from one of the
> interfaces).
>
> Reason being that if agent is on local host the executor running in its own
> network namespace will try to make a connection in its own network
> namespace and fail.
> On Wed, Aug 24, 2016 at 5:15 AM Olivier Sallou <ol...@irisa.fr>
> wrote:
>
> > I have the same behavior with Calico. Task get IP from CNI plugin, but
> > task remains in STAGING and same logs.
> >
> > mesos-execute --containerizer=mesos \
> > > --name=cni \
> > > --master=127.0.0.1:5050 \
> > > --networks=calico-net-1 \
> > > --command="ifconfig"
> > I0824 14:12:03.202328 24912 scheduler.cpp:172] Version: 1.0.0
> > I0824 14:12:03.203009 24911 scheduler.cpp:461] New master detected at
> > master@127.0.0.1:5050
> > Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0017'
> > Submitted task 'cni' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> >
> > REMAINS STAGING!
> >
> >
> > I0824 14:12:03.857158 24806 cni.cpp:1109] Got assigned IPv4 address
> > '192.168.0.0/32' from CNI network 'calico-net-1' for container
> > bdbb275a-ec5f-4a50-aca0-5e694ae57324
> > I0824 14:12:03.857348 24805 cni.cpp:838] Unable to find DNS nameservers
> > for container bdbb275a-ec5f-4a50-aca0-5e694ae57324. Using host
> > '/etc/resolv.conf'
> >
> > No more logs
> >
> >
> > Olivier
> >
> > On 08/24/2016 08:23 AM, Olivier Sallou wrote:
> > >
> > > On 08/23/2016 06:13 PM, Jie Yu wrote:
> > >> The DNS related logging means that the weave plugin does not return
> DNS
> > >> information, the agent uses the host resolv.conf for the container.
> So I
> > >> think is irrelevant to your problem.
> > >>
> > >> Mesos requires that executor can talk to agent. Can you see if there
> is
> > a
> > >> route from 10.32.0.1 to the agent IP?
> > > How can I check this as task does not start ? I have exposed weave
> > > network on host:
> > >
> > > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave
> expose
> > > 10.32.0.2
> > > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
> > > PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
> > > 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
> > > 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
> > > 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
> > > 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
> > >
> > > And why is it blocking?
> > >
> > > I am on a single host environement, so agent is on 127.0.0.1.
> > >
> > > Olivier
> > >> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <
> > olivier.sallou@irisa.fr>
> > >> wrote:
> > >>
> > >>> HI,
> > >>>
> > >>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
> > >>>
> > >>> Weave works nicely with the Docker containerizer.
> > >>>
> > >>> When I try to launch a task via my framework with unified
> > containerizer,
> > >>> the job remains waiting forever (no RUNNING message). I can see
> however
> > >>> that weave cni allocated an IP address to Mesos.
> > >>>
> > >>> I tried with a simple mesos-execute test.
> > >>>
> > >>> Example with a mesos-execute with no CNI, everything is OK
> > >>>
> > >>>
> > >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
> > mesos-execute
> > >>> --command="sleep 2" -docker_image=centos:latest --master=
> > 127.0.0.1:5050
> > >>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version:
> > 1.0.0
> > >>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
> > >>> master@127.0.0.1:5050
> > >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
> > >>> Submitted task 'test0' to agent
> > 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> > >>> Received status update TASK_RUNNING for task 'test0'
> > >>> source: SOURCE_EXECUTOR
> > >>> Received status update TASK_FINISHED for task 'test0'
> > >>> message: 'Command exited with status 0'
> > >>>
> > >>>
> > >>> Sample example specifying the weave network
> > >>>
> > >>>
> > >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
> > mesos-execute
> > >>> --command="sleep 2" -docker_image=centos:latest --master=
> > 127.0.0.1:5050
> > >>> --name=test0 --networks=weave
> > >>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
> > >>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
> > >>> master@127.0.0.1:5050
> > >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
> > >>> Submitted task 'test0' to agent
> > 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> > >>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
> > >>>
> > >>> mesos-slave logs:
> > >>>
> > >>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
> > >>> '/proc/28869/ns/net' to
> > >>>
> > '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-
> aa197646db09/ns'
> > >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> > >>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
> > >>> '10.32.0.1/12' from CNI network 'weave' for container
> > >>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> > >>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS
> nameservers
> > >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
> > >>> '/etc/resolv.conf'
> > >>>
> > >>> There are no other logs until I kill the job.
> > >>> We can see that Mesos container got an IP but it seems to block on
> DNS,
> > >>>
> > >>> Thanks for hints....
> > >>>
> > >>> --
> > >>>
> > >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >>>
> > >>>
> > >>>
> >
> > --
> > Olivier Sallou
> > IRISA / University of Rennes 1
> > Campus de Beaulieu, 35000 RENNES - FRANCE
> > Tel: 02.99.84.71.95
> >
> > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >
> >
>
Re: Fail to get CNI with unified containerizer, job remains stuck on staging
Posted by Avinash Sridharan <av...@mesosphere.io>.
Oliver, you can't have the agent running on 127.0.0.1. The agent needs to
be running in a routeabl IP address (choose an IP from one of the
interfaces).
Reason being that if agent is on local host the executor running in its own
network namespace will try to make a connection in its own network
namespace and fail.
On Wed, Aug 24, 2016 at 5:15 AM Olivier Sallou <ol...@irisa.fr>
wrote:
> I have the same behavior with Calico. Task get IP from CNI plugin, but
> task remains in STAGING and same logs.
>
> mesos-execute --containerizer=mesos \
> > --name=cni \
> > --master=127.0.0.1:5050 \
> > --networks=calico-net-1 \
> > --command="ifconfig"
> I0824 14:12:03.202328 24912 scheduler.cpp:172] Version: 1.0.0
> I0824 14:12:03.203009 24911 scheduler.cpp:461] New master detected at
> master@127.0.0.1:5050
> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0017'
> Submitted task 'cni' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>
> REMAINS STAGING!
>
>
> I0824 14:12:03.857158 24806 cni.cpp:1109] Got assigned IPv4 address
> '192.168.0.0/32' from CNI network 'calico-net-1' for container
> bdbb275a-ec5f-4a50-aca0-5e694ae57324
> I0824 14:12:03.857348 24805 cni.cpp:838] Unable to find DNS nameservers
> for container bdbb275a-ec5f-4a50-aca0-5e694ae57324. Using host
> '/etc/resolv.conf'
>
> No more logs
>
>
> Olivier
>
> On 08/24/2016 08:23 AM, Olivier Sallou wrote:
> >
> > On 08/23/2016 06:13 PM, Jie Yu wrote:
> >> The DNS related logging means that the weave plugin does not return DNS
> >> information, the agent uses the host resolv.conf for the container. So I
> >> think is irrelevant to your problem.
> >>
> >> Mesos requires that executor can talk to agent. Can you see if there is
> a
> >> route from 10.32.0.1 to the agent IP?
> > How can I check this as task does not start ? I have exposed weave
> > network on host:
> >
> > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
> > 10.32.0.2
> > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
> > PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
> > 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
> > 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
> > 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
> > 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
> >
> > And why is it blocking?
> >
> > I am on a single host environement, so agent is on 127.0.0.1.
> >
> > Olivier
> >> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> >> wrote:
> >>
> >>> HI,
> >>>
> >>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
> >>>
> >>> Weave works nicely with the Docker containerizer.
> >>>
> >>> When I try to launch a task via my framework with unified
> containerizer,
> >>> the job remains waiting forever (no RUNNING message). I can see however
> >>> that weave cni allocated an IP address to Mesos.
> >>>
> >>> I tried with a simple mesos-execute test.
> >>>
> >>> Example with a mesos-execute with no CNI, everything is OK
> >>>
> >>>
> >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
> mesos-execute
> >>> --command="sleep 2" -docker_image=centos:latest --master=
> 127.0.0.1:5050
> >>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version:
> 1.0.0
> >>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
> >>> master@127.0.0.1:5050
> >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
> >>> Submitted task 'test0' to agent
> 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> >>> Received status update TASK_RUNNING for task 'test0'
> >>> source: SOURCE_EXECUTOR
> >>> Received status update TASK_FINISHED for task 'test0'
> >>> message: 'Command exited with status 0'
> >>>
> >>>
> >>> Sample example specifying the weave network
> >>>
> >>>
> >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo
> mesos-execute
> >>> --command="sleep 2" -docker_image=centos:latest --master=
> 127.0.0.1:5050
> >>> --name=test0 --networks=weave
> >>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
> >>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
> >>> master@127.0.0.1:5050
> >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
> >>> Submitted task 'test0' to agent
> 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> >>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
> >>>
> >>> mesos-slave logs:
> >>>
> >>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
> >>> '/proc/28869/ns/net' to
> >>>
> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
> >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> >>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
> >>> '10.32.0.1/12' from CNI network 'weave' for container
> >>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> >>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
> >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
> >>> '/etc/resolv.conf'
> >>>
> >>> There are no other logs until I kill the job.
> >>> We can see that Mesos container got an IP but it seems to block on DNS,
> >>>
> >>> Thanks for hints....
> >>>
> >>> --
> >>>
> >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >>>
> >>>
> >>>
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: Fail to get CNI with unified containerizer, job remains stuck on
staging
Posted by Olivier Sallou <ol...@irisa.fr>.
I have the same behavior with Calico. Task get IP from CNI plugin, but
task remains in STAGING and same logs.
mesos-execute --containerizer=mesos \
> --name=cni \
> --master=127.0.0.1:5050 \
> --networks=calico-net-1 \
> --command="ifconfig"
I0824 14:12:03.202328 24912 scheduler.cpp:172] Version: 1.0.0
I0824 14:12:03.203009 24911 scheduler.cpp:461] New master detected at
master@127.0.0.1:5050
Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0017'
Submitted task 'cni' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
REMAINS STAGING!
I0824 14:12:03.857158 24806 cni.cpp:1109] Got assigned IPv4 address
'192.168.0.0/32' from CNI network 'calico-net-1' for container
bdbb275a-ec5f-4a50-aca0-5e694ae57324
I0824 14:12:03.857348 24805 cni.cpp:838] Unable to find DNS nameservers
for container bdbb275a-ec5f-4a50-aca0-5e694ae57324. Using host
'/etc/resolv.conf'
No more logs
Olivier
On 08/24/2016 08:23 AM, Olivier Sallou wrote:
>
> On 08/23/2016 06:13 PM, Jie Yu wrote:
>> The DNS related logging means that the weave plugin does not return DNS
>> information, the agent uses the host resolv.conf for the container. So I
>> think is irrelevant to your problem.
>>
>> Mesos requires that executor can talk to agent. Can you see if there is a
>> route from 10.32.0.1 to the agent IP?
> How can I check this as task does not start ? I have exposed weave
> network on host:
>
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
> 10.32.0.2
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
> PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
> 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
> 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
>
> And why is it blocking?
>
> I am on a single host environement, so agent is on 127.0.0.1.
>
> Olivier
>> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <ol...@irisa.fr>
>> wrote:
>>
>>> HI,
>>>
>>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>>
>>> Weave works nicely with the Docker containerizer.
>>>
>>> When I try to launch a task via my framework with unified containerizer,
>>> the job remains waiting forever (no RUNNING message). I can see however
>>> that weave cni allocated an IP address to Mesos.
>>>
>>> I tried with a simple mesos-execute test.
>>>
>>> Example with a mesos-execute with no CNI, everything is OK
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> Received status update TASK_RUNNING for task 'test0'
>>> source: SOURCE_EXECUTOR
>>> Received status update TASK_FINISHED for task 'test0'
>>> message: 'Command exited with status 0'
>>>
>>>
>>> Sample example specifying the weave network
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0 --networks=weave
>>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>>
>>> mesos-slave logs:
>>>
>>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>>> '/proc/28869/ns/net' to
>>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>>> '10.32.0.1/12' from CNI network 'weave' for container
>>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>>> '/etc/resolv.conf'
>>>
>>> There are no other logs until I kill the job.
>>> We can see that Mesos container got an IP but it seems to block on DNS,
>>>
>>> Thanks for hints....
>>>
>>> --
>>>
>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>
>>>
>>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to get CNI with unified containerizer, job remains stuck on
staging
Posted by Olivier Sallou <ol...@irisa.fr>.
On 08/24/2016 08:23 AM, Olivier Sallou wrote:
>
> On 08/23/2016 06:13 PM, Jie Yu wrote:
>> The DNS related logging means that the weave plugin does not return DNS
>> information, the agent uses the host resolv.conf for the container. So I
>> think is irrelevant to your problem.
>>
>> Mesos requires that executor can talk to agent. Can you see if there is a
>> route from 10.32.0.1 to the agent IP?
> How can I check this as task does not start ? I have exposed weave
> network on host:
>
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
> 10.32.0.2
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
> PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
> 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
> 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
>
> And why is it blocking?
>
> I am on a single host environement, so agent is on 127.0.0.1.
By the way, running a Docker container to use the weave CNI plugin works
fine, it gets it IP and container runs nicely.
>
> Olivier
>> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <ol...@irisa.fr>
>> wrote:
>>
>>> HI,
>>>
>>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>>
>>> Weave works nicely with the Docker containerizer.
>>>
>>> When I try to launch a task via my framework with unified containerizer,
>>> the job remains waiting forever (no RUNNING message). I can see however
>>> that weave cni allocated an IP address to Mesos.
>>>
>>> I tried with a simple mesos-execute test.
>>>
>>> Example with a mesos-execute with no CNI, everything is OK
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> Received status update TASK_RUNNING for task 'test0'
>>> source: SOURCE_EXECUTOR
>>> Received status update TASK_FINISHED for task 'test0'
>>> message: 'Command exited with status 0'
>>>
>>>
>>> Sample example specifying the weave network
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0 --networks=weave
>>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>>
>>> mesos-slave logs:
>>>
>>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>>> '/proc/28869/ns/net' to
>>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>>> '10.32.0.1/12' from CNI network 'weave' for container
>>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>>> '/etc/resolv.conf'
>>>
>>> There are no other logs until I kill the job.
>>> We can see that Mesos container got an IP but it seems to block on DNS,
>>>
>>> Thanks for hints....
>>>
>>> --
>>>
>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>
>>>
>>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to get CNI with unified containerizer, job remains stuck on
staging
Posted by Olivier Sallou <ol...@irisa.fr>.
On 08/23/2016 06:13 PM, Jie Yu wrote:
> The DNS related logging means that the weave plugin does not return DNS
> information, the agent uses the host resolv.conf for the container. So I
> think is irrelevant to your problem.
>
> Mesos requires that executor can talk to agent. Can you see if there is a
> route from 10.32.0.1 to the agent IP?
How can I check this as task does not start ? I have exposed weave
network on host:
osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
10.32.0.2
osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
And why is it blocking?
I am on a single host environement, so agent is on 127.0.0.1.
Olivier
>
> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>> HI,
>>
>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>
>> Weave works nicely with the Docker containerizer.
>>
>> When I try to launch a task via my framework with unified containerizer,
>> the job remains waiting forever (no RUNNING message). I can see however
>> that weave cni allocated an IP address to Mesos.
>>
>> I tried with a simple mesos-execute test.
>>
>> Example with a mesos-execute with no CNI, everything is OK
>>
>>
>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>> master@127.0.0.1:5050
>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>> Received status update TASK_RUNNING for task 'test0'
>> source: SOURCE_EXECUTOR
>> Received status update TASK_FINISHED for task 'test0'
>> message: 'Command exited with status 0'
>>
>>
>> Sample example specifying the weave network
>>
>>
>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>> --name=test0 --networks=weave
>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>> master@127.0.0.1:5050
>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>
>> mesos-slave logs:
>>
>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>> '/proc/28869/ns/net' to
>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>> '10.32.0.1/12' from CNI network 'weave' for container
>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>> '/etc/resolv.conf'
>>
>> There are no other logs until I kill the job.
>> We can see that Mesos container got an IP but it seems to block on DNS,
>>
>> Thanks for hints....
>>
>> --
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to get CNI with unified containerizer, job remains stuck on staging
Posted by Jie Yu <yu...@gmail.com>.
The DNS related logging means that the weave plugin does not return DNS
information, the agent uses the host resolv.conf for the container. So I
think is irrelevant to your problem.
Mesos requires that executor can talk to agent. Can you see if there is a
route from 10.32.0.1 to the agent IP?
On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
> HI,
>
> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>
> Weave works nicely with the Docker containerizer.
>
> When I try to launch a task via my framework with unified containerizer,
> the job remains waiting forever (no RUNNING message). I can see however
> that weave cni allocated an IP address to Mesos.
>
> I tried with a simple mesos-execute test.
>
> Example with a mesos-execute with no CNI, everything is OK
>
>
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
> master@127.0.0.1:5050
> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> Received status update TASK_RUNNING for task 'test0'
> source: SOURCE_EXECUTOR
> Received status update TASK_FINISHED for task 'test0'
> message: 'Command exited with status 0'
>
>
> Sample example specifying the weave network
>
>
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
> --name=test0 --networks=weave
> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
> master@127.0.0.1:5050
> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>
> mesos-slave logs:
>
> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
> '/proc/28869/ns/net' to
> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
> '10.32.0.1/12' from CNI network 'weave' for container
> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
> '/etc/resolv.conf'
>
> There are no other logs until I kill the job.
> We can see that Mesos container got an IP but it seems to block on DNS,
>
> Thanks for hints....
>
> --
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
>