You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Gmail <el...@gmail.com> on 2016/09/27 12:08:54 UTC

Framework is registered, but never receives resource offers

Hi

I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui. 

I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.

Anyone have an idea what the problem might be?

One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.

Is this a possibility?

Thanks
Eli

Re: Framework is registered, but never receives resource offers

Posted by Eli Jordan <el...@gmail.com>.
I haven't yet looked at the logs for the mesos master when this issue happens. Just the logs of my framework. I will investigate further.

However, about 70% of the time everything works just fine. So I'm not sure if this is the issue, unless one of the containers didn't get configured correctly by minimesos.

I also run the scheduler in a container, and use docker-java to start it in my tests. 

Thanks
Eli

> On 28 Sep. 2016, at 02:00, Hendrik Haddorp <he...@gmx.net> wrote:
> 
> On mini mesos you might have the same problem, like libprocess might bind to the loopback device. The other tricky thing is what the individual docker containers can reach and by what host name then can. I think my setup is quite similar I just opted to not use mini mesos and created something quite similar. In that setup I create a docker network for my containers so that they can see each other by name. the driver for my framework is also running in a container so that I don't need the native library on my system. This also has the effect that libprocess should have no problems with the communication as it is all using IPs from the same network.
> 
> You might also want to check the master log. In my case I saw a framework registration immediately followed by a disconnect. I believe this indicates that there is a communication issue.
> 
>> On 27.09.2016 15:09, Gmail wrote:
>> Thanks Hendrik
>> 
>> I have solved that particular problem, when running a different framework in docker. It was a bit of a challenge to get the right incantation of environment variables and ports defined, but is working reliably now.
>> 
>> I mainly hit this issue when running my integration tests, where I also run the mesos master and agent in docker using mini mesos
>> 
>> Sent from my iPad
>> 
>>> On 27 Sep 2016, at 22:51, Hendrik Haddorp <he...@gmx.net> wrote:
>>> 
>>> Hi,
>>> 
>>> this sounds quite like a problem I had hit a few days ago. If you are using the mesos native library you need to make sure that the LIBPROCESS environment variables are set correctly. Otherwise the Mesos master can not communicate back to your process, especially if you are not running on the same node as the master. Things gets slightly more tricky if your scheduler is running in a docker container.
>>> 
>>> regards,
>>> Hendrik
>>> 
>>>> On 27.09.2016 14:34, Eli Jordan wrote:
>>>> Yes, it appears in the mesos ui, and stays there. I log all messages from the mesos master, including resource offers and disconnected. I don't receive offers or disconnected.
>>>> 
>>>> I know I need to accept or decline the offers, the problem is that I never receive the resource offer, but the master thinks I have.
>>>> 
>>>> This only happens sometimes, sometimes the framework starts just fine, and can launch tasks. Which is what led me to think it might be a timing issue.
>>>> 
>>>> Thanks
>>>> Eli
>>>> 
>>>>> On 27 Sep. 2016, at 22:25, Olivier Sallou <olivier.sallou@irisa.fr <ma...@irisa.fr>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 09/27/2016 02:08 PM, Gmail wrote:
>>>>>> Hi
>>>>>> 
>>>>>> I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui.
>>>>>> 
>>>>>> I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.
>>>>> Does your framework appear in mesos UI in the list fo frameworks ? (and
>>>>> remains in the list)
>>>>> 
>>>>> Maybe your framework is registered then disconnected.
>>>>>> Anyone have an idea what the problem might be?
>>>>>> 
>>>>>> One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.
>>>>> Do you log when you received offers? When you receive an offer you must
>>>>> accept or decline the offers.
>>>>> 
>>>>> Olivier
>>>>>> Is this a possibility?
>>>>>> 
>>>>>> Thanks
>>>>>> Eli
>>>>> -- 
>>>>> Olivier Sallou
>>>>> IRISA / University of Rennes 1
>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>> Tel: 02.99.84.71.95
>>>>> 
>>>>> gpg key id: 4096R/326D8438  (keyring.debian.org <http://keyring.debian.org>)
>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> 

Re: Framework is registered, but never receives resource offers

Posted by Hendrik Haddorp <he...@gmx.net>.
On mini mesos you might have the same problem, like libprocess might 
bind to the loopback device. The other tricky thing is what the 
individual docker containers can reach and by what host name then can. I 
think my setup is quite similar I just opted to not use mini mesos and 
created something quite similar. In that setup I create a docker network 
for my containers so that they can see each other by name. the driver 
for my framework is also running in a container so that I don't need the 
native library on my system. This also has the effect that libprocess 
should have no problems with the communication as it is all using IPs 
from the same network.

You might also want to check the master log. In my case I saw a 
framework registration immediately followed by a disconnect. I believe 
this indicates that there is a communication issue.

On 27.09.2016 15:09, Gmail wrote:
> Thanks Hendrik
>
> I have solved that particular problem, when running a different framework in docker. It was a bit of a challenge to get the right incantation of environment variables and ports defined, but is working reliably now.
>
> I mainly hit this issue when running my integration tests, where I also run the mesos master and agent in docker using mini mesos
>
> Sent from my iPad
>
>> On 27 Sep 2016, at 22:51, Hendrik Haddorp <he...@gmx.net> wrote:
>>
>> Hi,
>>
>> this sounds quite like a problem I had hit a few days ago. If you are using the mesos native library you need to make sure that the LIBPROCESS environment variables are set correctly. Otherwise the Mesos master can not communicate back to your process, especially if you are not running on the same node as the master. Things gets slightly more tricky if your scheduler is running in a docker container.
>>
>> regards,
>> Hendrik
>>
>>> On 27.09.2016 14:34, Eli Jordan wrote:
>>> Yes, it appears in the mesos ui, and stays there. I log all messages from the mesos master, including resource offers and disconnected. I don't receive offers or disconnected.
>>>
>>> I know I need to accept or decline the offers, the problem is that I never receive the resource offer, but the master thinks I have.
>>>
>>> This only happens sometimes, sometimes the framework starts just fine, and can launch tasks. Which is what led me to think it might be a timing issue.
>>>
>>> Thanks
>>> Eli
>>>
>>>> On 27 Sep. 2016, at 22:25, Olivier Sallou <olivier.sallou@irisa.fr <ma...@irisa.fr>> wrote:
>>>>
>>>>
>>>>
>>>>> On 09/27/2016 02:08 PM, Gmail wrote:
>>>>> Hi
>>>>>
>>>>> I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui.
>>>>>
>>>>> I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.
>>>> Does your framework appear in mesos UI in the list fo frameworks ? (and
>>>> remains in the list)
>>>>
>>>> Maybe your framework is registered then disconnected.
>>>>> Anyone have an idea what the problem might be?
>>>>>
>>>>> One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.
>>>> Do you log when you received offers? When you receive an offer you must
>>>> accept or decline the offers.
>>>>
>>>> Olivier
>>>>> Is this a possibility?
>>>>>
>>>>> Thanks
>>>>> Eli
>>>> -- 
>>>> Olivier Sallou
>>>> IRISA / University of Rennes 1
>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>> Tel: 02.99.84.71.95
>>>>
>>>> gpg key id: 4096R/326D8438  (keyring.debian.org <http://keyring.debian.org>)
>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438


Re: Framework is registered, but never receives resource offers

Posted by Gmail <el...@gmail.com>.
Thanks Hendrik

I have solved that particular problem, when running a different framework in docker. It was a bit of a challenge to get the right incantation of environment variables and ports defined, but is working reliably now.

I mainly hit this issue when running my integration tests, where I also run the mesos master and agent in docker using mini mesos

Sent from my iPad

> On 27 Sep 2016, at 22:51, Hendrik Haddorp <he...@gmx.net> wrote:
> 
> Hi,
> 
> this sounds quite like a problem I had hit a few days ago. If you are using the mesos native library you need to make sure that the LIBPROCESS environment variables are set correctly. Otherwise the Mesos master can not communicate back to your process, especially if you are not running on the same node as the master. Things gets slightly more tricky if your scheduler is running in a docker container.
> 
> regards,
> Hendrik
> 
>> On 27.09.2016 14:34, Eli Jordan wrote:
>> Yes, it appears in the mesos ui, and stays there. I log all messages from the mesos master, including resource offers and disconnected. I don't receive offers or disconnected.
>> 
>> I know I need to accept or decline the offers, the problem is that I never receive the resource offer, but the master thinks I have.
>> 
>> This only happens sometimes, sometimes the framework starts just fine, and can launch tasks. Which is what led me to think it might be a timing issue.
>> 
>> Thanks
>> Eli
>> 
>>> On 27 Sep. 2016, at 22:25, Olivier Sallou <olivier.sallou@irisa.fr <ma...@irisa.fr>> wrote:
>>> 
>>> 
>>> 
>>>> On 09/27/2016 02:08 PM, Gmail wrote:
>>>> Hi
>>>> 
>>>> I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui.
>>>> 
>>>> I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.
>>> Does your framework appear in mesos UI in the list fo frameworks ? (and
>>> remains in the list)
>>> 
>>> Maybe your framework is registered then disconnected.
>>>> 
>>>> Anyone have an idea what the problem might be?
>>>> 
>>>> One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.
>>> Do you log when you received offers? When you receive an offer you must
>>> accept or decline the offers.
>>> 
>>> Olivier
>>>> 
>>>> Is this a possibility?
>>>> 
>>>> Thanks
>>>> Eli
>>> 
>>> -- 
>>> Olivier Sallou
>>> IRISA / University of Rennes 1
>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>> Tel: 02.99.84.71.95
>>> 
>>> gpg key id: 4096R/326D8438  (keyring.debian.org <http://keyring.debian.org>)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> 

Re: Framework is registered, but never receives resource offers

Posted by Hendrik Haddorp <he...@gmx.net>.
Hi,

this sounds quite like a problem I had hit a few days ago. If you are 
using the mesos native library you need to make sure that the LIBPROCESS 
environment variables are set correctly. Otherwise the Mesos master can 
not communicate back to your process, especially if you are not running 
on the same node as the master. Things gets slightly more tricky if your 
scheduler is running in a docker container.

regards,
Hendrik

On 27.09.2016 14:34, Eli Jordan wrote:
> Yes, it appears in the mesos ui, and stays there. I log all messages 
> from the mesos master, including resource offers and disconnected. I 
> don't receive offers or disconnected.
>
> I know I need to accept or decline the offers, the problem is that I 
> never receive the resource offer, but the master thinks I have.
>
> This only happens sometimes, sometimes the framework starts just fine, 
> and can launch tasks. Which is what led me to think it might be a 
> timing issue.
>
> Thanks
> Eli
>
> On 27 Sep. 2016, at 22:25, Olivier Sallou <olivier.sallou@irisa.fr 
> <ma...@irisa.fr>> wrote:
>
>>
>>
>> On 09/27/2016 02:08 PM, Gmail wrote:
>>> Hi
>>>
>>> I am implementing a mesos framework, and have hit a strange issue 
>>> that I can't make sense of. Intermittently, my framework will 
>>> receive the registered message, and is shown as registered in the 
>>> mesos ui.
>>>
>>> I never see any resource offer messages being processed by the 
>>> framework, however, the mesos master indicates that it has offered 
>>> resources to the framework (on the frameworks page in the ui). In 
>>> this case, I only have one slave, and all the resources are 
>>> apparently being consumed by the framework, so no tasks can be launched.
>> Does your framework appear in mesos UI in the list fo frameworks ? (and
>> remains in the list)
>>
>> Maybe your framework is registered then disconnected.
>>>
>>> Anyone have an idea what the problem might be?
>>>
>>> One thought I had, is that the MesosSchedulerDriver isn't expecting 
>>> the scheduler implementation to process messages asynchronously, but 
>>> I couldn't find any documentation indicating one way or the other. 
>>> In my case, I'm using akka actors, and all the scheduler 
>>> implementation does is dispatch a message.
>> Do you log when you received offers? When you receive an offer you must
>> accept or decline the offers.
>>
>> Olivier
>>>
>>> Is this a possibility?
>>>
>>> Thanks
>>> Eli
>>
>> -- 
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438  (keyring.debian.org 
>> <http://keyring.debian.org>)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>>


Re: Framework is registered, but never receives resource offers

Posted by Eli Jordan <el...@gmail.com>.
Yes, it appears in the mesos ui, and stays there. I log all messages from the mesos master, including resource offers and disconnected. I don't receive offers or disconnected.

I know I need to accept or decline the offers, the problem is that I never receive the resource offer, but the master thinks I have.

This only happens sometimes, sometimes the framework starts just fine, and can launch tasks. Which is what led me to think it might be a timing issue.

Thanks
Eli

> On 27 Sep. 2016, at 22:25, Olivier Sallou <ol...@irisa.fr> wrote:
> 
> 
> 
>> On 09/27/2016 02:08 PM, Gmail wrote:
>> Hi
>> 
>> I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui. 
>> 
>> I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.
> Does your framework appear in mesos UI in the list fo frameworks ? (and
> remains in the list)
> 
> Maybe your framework is registered then disconnected.
>> 
>> Anyone have an idea what the problem might be?
>> 
>> One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.
> Do you log when you received offers? When you receive an offer you must
> accept or decline the offers.
> 
> Olivier
>> 
>> Is this a possibility?
>> 
>> Thanks
>> Eli
> 
> -- 
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
> 
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> 

Re: Framework is registered, but never receives resource offers

Posted by Olivier Sallou <ol...@irisa.fr>.

On 09/27/2016 02:08 PM, Gmail wrote:
> Hi
>
> I am implementing a mesos framework, and have hit a strange issue that I can't make sense of. Intermittently, my framework will receive the registered message, and is shown as registered in the mesos ui. 
>
> I never see any resource offer messages being processed by the framework, however, the mesos master indicates that it has offered resources to the framework (on the frameworks page in the ui). In this case, I only have one slave, and all the resources are apparently being consumed by the framework, so no tasks can be launched.
Does your framework appear in mesos UI in the list fo frameworks ? (and
remains in the list)

Maybe your framework is registered then disconnected.
>
> Anyone have an idea what the problem might be?
>
> One thought I had, is that the MesosSchedulerDriver isn't expecting the scheduler implementation to process messages asynchronously, but I couldn't find any documentation indicating one way or the other. In my case, I'm using akka actors, and all the scheduler implementation does is dispatch a message.
Do you log when you received offers? When you receive an offer you must
accept or decline the offers.

Olivier
>
> Is this a possibility?
>
> Thanks
> Eli

-- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438