You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by canan chen <cc...@gmail.com> on 2015/09/08 09:24:59 UTC
Can not allocate executor when running spark on mesos
Hi all,
I try to run spark on mesos, but it looks like I can not allocate resources
from mesos. I am not expert of mesos, but from the mesos log, it seems
spark always decline the offer from mesos. Not sure what's wrong, maybe
need some configuration change. Here's the mesos master log
I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
request for framework 'Spark shell' at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
20150908-143320-16777343-5050-41965-0000 (Spark shell) at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
checkpointing disabled and capabilities [ ]
I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
20150908-143320-16777343-5050-41965-0000
I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE call
for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
20150908-143320-16777343-5050-41965-0000 (Spark shell) at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered cpus(*):8;
mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total: cpus(*):8;
mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated: ) on slave
20150908-143320-16777343-5050-41965-S0 from framework
20150908-143320-16777343-5050-41965-0000
I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE call
for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
20150908-143320-16777343-5050-41965-0000 (Spark shell) at
scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
Re: Can not allocate executor when running spark on mesos
Posted by Iulian Dragoș <iu...@typesafe.com>.
On Thu, Sep 10, 2015 at 3:35 AM, canan chen <cc...@gmail.com> wrote:
> Finally got the answer. Actually it works fine. The allocation behavior
> on mesos is a little different from yarn/standalone. Seems the executor in
> mesos is lazily allocated (only when job is executed) while executor in
> yarn/standalone is allocated when spark-shell is started.
>
That's in fine-grained mode (the default). You can turn on coarse-grained
mode to acquire executors on startup.
iulian
>
>
>
> On Tue, Sep 8, 2015 at 10:39 PM, canan chen <cc...@gmail.com> wrote:
>
>> Yes, I follow the guide in this doc, and run it as mesos client mode
>>
>> On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> In which mode are you submitting your application? (coarse-grained or
>>> fine-grained(default)). Have you gone through this documentation already?
>>> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Tue, Sep 8, 2015 at 12:54 PM, canan chen <cc...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I try to run spark on mesos, but it looks like I can not allocate
>>>> resources from mesos. I am not expert of mesos, but from the mesos log, it
>>>> seems spark always decline the offer from mesos. Not sure what's wrong,
>>>> maybe need some configuration change. Here's the mesos master log
>>>>
>>>> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
>>>> request for framework 'Spark shell' at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
>>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
>>>> checkpointing disabled and capabilities [ ]
>>>> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
>>>> 20150908-143320-16777343-5050-41965-0000
>>>> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
>>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE
>>>> call for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
>>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered
>>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total:
>>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated:
>>>> ) on slave 20150908-143320-16777343-5050-41965-S0 from framework
>>>> 20150908-143320-16777343-5050-41965-0000
>>>> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
>>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE
>>>> call for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
>>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>>
>>>
>>>
>>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Can not allocate executor when running spark on mesos
Posted by canan chen <cc...@gmail.com>.
Finally got the answer. Actually it works fine. The allocation behavior on
mesos is a little different from yarn/standalone. Seems the executor in
mesos is lazily allocated (only when job is executed) while executor in
yarn/standalone is allocated when spark-shell is started.
On Tue, Sep 8, 2015 at 10:39 PM, canan chen <cc...@gmail.com> wrote:
> Yes, I follow the guide in this doc, and run it as mesos client mode
>
> On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> In which mode are you submitting your application? (coarse-grained or
>> fine-grained(default)). Have you gone through this documentation already?
>> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Sep 8, 2015 at 12:54 PM, canan chen <cc...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I try to run spark on mesos, but it looks like I can not allocate
>>> resources from mesos. I am not expert of mesos, but from the mesos log, it
>>> seems spark always decline the offer from mesos. Not sure what's wrong,
>>> maybe need some configuration change. Here's the mesos master log
>>>
>>> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
>>> request for framework 'Spark shell' at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
>>> checkpointing disabled and capabilities [ ]
>>> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
>>> 20150908-143320-16777343-5050-41965-0000
>>> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE call
>>> for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered
>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total:
>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated:
>>> ) on slave 20150908-143320-16777343-5050-41965-S0 from framework
>>> 20150908-143320-16777343-5050-41965-0000
>>> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE call
>>> for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>>
>>
>>
>
Re: Can not allocate executor when running spark on mesos
Posted by canan chen <cc...@gmail.com>.
Yes, I follow the guide in this doc, and run it as mesos client mode
On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> In which mode are you submitting your application? (coarse-grained or
> fine-grained(default)). Have you gone through this documentation already?
> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url
>
> Thanks
> Best Regards
>
> On Tue, Sep 8, 2015 at 12:54 PM, canan chen <cc...@gmail.com> wrote:
>
>> Hi all,
>>
>> I try to run spark on mesos, but it looks like I can not allocate
>> resources from mesos. I am not expert of mesos, but from the mesos log, it
>> seems spark always decline the offer from mesos. Not sure what's wrong,
>> maybe need some configuration change. Here's the mesos master log
>>
>> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
>> request for framework 'Spark shell' at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
>> checkpointing disabled and capabilities [ ]
>> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
>> 20150908-143320-16777343-5050-41965-0000
>> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE call
>> for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered
>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total:
>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated:
>> ) on slave 20150908-143320-16777343-5050-41965-S0 from framework
>> 20150908-143320-16777343-5050-41965-0000
>> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE call
>> for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>>
>
>
Re: Can not allocate executor when running spark on mesos
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
In which mode are you submitting your application? (coarse-grained or
fine-grained(default)). Have you gone through this documentation already?
http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url
Thanks
Best Regards
On Tue, Sep 8, 2015 at 12:54 PM, canan chen <cc...@gmail.com> wrote:
> Hi all,
>
> I try to run spark on mesos, but it looks like I can not allocate
> resources from mesos. I am not expert of mesos, but from the mesos log, it
> seems spark always decline the offer from mesos. Not sure what's wrong,
> maybe need some configuration change. Here's the mesos master log
>
> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
> request for framework 'Spark shell' at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
> checkpointing disabled and capabilities [ ]
> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
> 20150908-143320-16777343-5050-41965-0000
> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE call
> for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered cpus(*):8;
> mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total: cpus(*):8;
> mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated: ) on slave
> 20150908-143320-16777343-5050-41965-S0 from framework
> 20150908-143320-16777343-5050-41965-0000
> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE call
> for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at
> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
>