You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Peter Figliozzi <pe...@gmail.com> on 2016/08/14 21:37:15 UTC
resources in Mesos UI but none given to Spark
Hi All, I am new to Mesos. I set up a cluster this weekend with 3 agents,
1 master, Mesos 1.0.0. The resources show in the Mesos UI and the agents
are all in the Agents tab. So everything looks good from that vantage
point.
Next I installed Spark 2.0.0 on each agent and the master, in the same path
(/opt/spark) on each machine. I run the spark-shell from the master like
this:
./spark-shell --master mesos://zk://moe:2181/mesos -c
spark.mesos.executor.home=`pwd`
The shell comes up nicely, however, none of the resources get assigned to
the Spark framework (zeros for everything).
If I try a simple task like
sc.parallelize(0 to 10, 8).count
it fails:
WARN TaskSchedulerImpl: Initial job has not accepted any resources; check
your cluster UI to ensure that workers are registered and have sufficient
resources
I'll post my logs in a little bit if need be. Hopefully it's a common newb
error and simple fix.
Thank you
Re: resources in Mesos UI but none given to Spark
Posted by Peter Figliozzi <pe...@gmail.com>.
Problem solved-- I was incorrectly specifying the spark directory in the
command-line argument. It wants the spark root directory, not the bin.
I unpacked mine in /opt, so, the Right way:
spark.mesos.executor.home=/opt/spark-2.0.0-bin-hadoop2.7/
Wrong way:
spark.mesos.executor.home=/opt/spark-2.0.0-bin-hadoop2.7/bin
On Sun, Aug 14, 2016 at 9:09 PM, Peter Figliozzi <pe...@gmail.com>
wrote:
>
> I notice the first thing that happens when I run the spark-shell is *three
> failed 'sandbox' tasks* appearing in the Mesos UI. (One for each agent.)
> Then another three, as if it tried twice.
>
> I attached the Mesos logs from the master and one of the agents.
>
> All of this happens when I run the spark-shell... before any commands in
> the shell.
>
> Summary/Master:
>
> - Received SUBSCRIBE call for spark shell
> - Adds and launches task with 4 cpus and 1408 mem on each agent
> - Something about "Processing ACCEPT call for offers"
> - TASK_RUNNING on all agents
> - Processing ACKNOWLEDGE calls all agents
> - Status update TASK_FAILED from all agents
>
> Summary/Agent:
>
> - "Got assigned" task 1
> - Trying to chown /var/lib/mesos/slaves/..... to user 'peter'
> - Launching executor 1 of framework with resources cpus 0.1, mem 32 in
> work directory...
> - Queuing task '1' for executor '1'...
> - docker: No container info found, skipping launch
> - containerizer: starting container...
> - linux_launcher: cloning child process with flags =
> - systemd: assigned child process '17043' to 'mesos_executors.slice'
> - ... some more slave stuff ....
> - Handling status update TASK_FAILED for task 1
>
> Could someone explain what's supposed to happen here when running the
> spark shell?
>
> Thanks,
>
> Pete
>
>
> On Sun, Aug 14, 2016 at 6:57 PM, Michael Gummelt <mg...@mesosphere.io>
> wrote:
>
>> Turning on Spark debug logs in conf/log4j.properties may help. The
>> problem could be any number of things, including that you don't have enough
>> resources for the default executor sizes.
>>
>> On Sun, Aug 14, 2016 at 2:37 PM, Peter Figliozzi <
>> pete.figliozzi@gmail.com> wrote:
>>
>>> Hi All, I am new to Mesos. I set up a cluster this weekend with 3
>>> agents, 1 master, Mesos 1.0.0. The resources show in the Mesos UI and the
>>> agents are all in the Agents tab. So everything looks good from that
>>> vantage point.
>>>
>>> Next I installed Spark 2.0.0 on each agent and the master, in the same
>>> path (/opt/spark) on each machine. I run the spark-shell from the master
>>> like this:
>>>
>>> ./spark-shell --master mesos://zk://moe:2181/mesos -c
>>> spark.mesos.executor.home=`pwd`
>>>
>>> The shell comes up nicely, however, none of the resources get assigned
>>> to the Spark framework (zeros for everything).
>>>
>>> If I try a simple task like
>>>
>>> sc.parallelize(0 to 10, 8).count
>>>
>>> it fails:
>>>
>>> WARN TaskSchedulerImpl: Initial job has not accepted any resources;
>>> check your cluster UI to ensure that workers are registered and have
>>> sufficient resources
>>>
>>>
>>> I'll post my logs in a little bit if need be. Hopefully it's a common
>>> newb error and simple fix.
>>>
>>> Thank you
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>>
>
>
Re: resources in Mesos UI but none given to Spark
Posted by Peter Figliozzi <pe...@gmail.com>.
I notice the first thing that happens when I run the spark-shell is *three
failed 'sandbox' tasks* appearing in the Mesos UI. (One for each agent.)
Then another three, as if it tried twice.
I attached the Mesos logs from the master and one of the agents.
All of this happens when I run the spark-shell... before any commands in
the shell.
Summary/Master:
- Received SUBSCRIBE call for spark shell
- Adds and launches task with 4 cpus and 1408 mem on each agent
- Something about "Processing ACCEPT call for offers"
- TASK_RUNNING on all agents
- Processing ACKNOWLEDGE calls all agents
- Status update TASK_FAILED from all agents
Summary/Agent:
- "Got assigned" task 1
- Trying to chown /var/lib/mesos/slaves/..... to user 'peter'
- Launching executor 1 of framework with resources cpus 0.1, mem 32 in
work directory...
- Queuing task '1' for executor '1'...
- docker: No container info found, skipping launch
- containerizer: starting container...
- linux_launcher: cloning child process with flags =
- systemd: assigned child process '17043' to 'mesos_executors.slice'
- ... some more slave stuff ....
- Handling status update TASK_FAILED for task 1
Could someone explain what's supposed to happen here when running the spark
shell?
Thanks,
Pete
On Sun, Aug 14, 2016 at 6:57 PM, Michael Gummelt <mg...@mesosphere.io>
wrote:
> Turning on Spark debug logs in conf/log4j.properties may help. The
> problem could be any number of things, including that you don't have enough
> resources for the default executor sizes.
>
> On Sun, Aug 14, 2016 at 2:37 PM, Peter Figliozzi <pete.figliozzi@gmail.com
> > wrote:
>
>> Hi All, I am new to Mesos. I set up a cluster this weekend with 3
>> agents, 1 master, Mesos 1.0.0. The resources show in the Mesos UI and the
>> agents are all in the Agents tab. So everything looks good from that
>> vantage point.
>>
>> Next I installed Spark 2.0.0 on each agent and the master, in the same
>> path (/opt/spark) on each machine. I run the spark-shell from the master
>> like this:
>>
>> ./spark-shell --master mesos://zk://moe:2181/mesos -c
>> spark.mesos.executor.home=`pwd`
>>
>> The shell comes up nicely, however, none of the resources get assigned to
>> the Spark framework (zeros for everything).
>>
>> If I try a simple task like
>>
>> sc.parallelize(0 to 10, 8).count
>>
>> it fails:
>>
>> WARN TaskSchedulerImpl: Initial job has not accepted any resources; check
>> your cluster UI to ensure that workers are registered and have sufficient
>> resources
>>
>>
>> I'll post my logs in a little bit if need be. Hopefully it's a common
>> newb error and simple fix.
>>
>> Thank you
>>
>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>
Re: resources in Mesos UI but none given to Spark
Posted by Michael Gummelt <mg...@mesosphere.io>.
Turning on Spark debug logs in conf/log4j.properties may help. The problem
could be any number of things, including that you don't have enough
resources for the default executor sizes.
On Sun, Aug 14, 2016 at 2:37 PM, Peter Figliozzi <pe...@gmail.com>
wrote:
> Hi All, I am new to Mesos. I set up a cluster this weekend with 3 agents,
> 1 master, Mesos 1.0.0. The resources show in the Mesos UI and the agents
> are all in the Agents tab. So everything looks good from that vantage
> point.
>
> Next I installed Spark 2.0.0 on each agent and the master, in the same
> path (/opt/spark) on each machine. I run the spark-shell from the master
> like this:
>
> ./spark-shell --master mesos://zk://moe:2181/mesos -c
> spark.mesos.executor.home=`pwd`
>
> The shell comes up nicely, however, none of the resources get assigned to
> the Spark framework (zeros for everything).
>
> If I try a simple task like
>
> sc.parallelize(0 to 10, 8).count
>
> it fails:
>
> WARN TaskSchedulerImpl: Initial job has not accepted any resources; check
> your cluster UI to ensure that workers are registered and have sufficient
> resources
>
>
> I'll post my logs in a little bit if need be. Hopefully it's a common
> newb error and simple fix.
>
> Thank you
>
--
Michael Gummelt
Software Engineer
Mesosphere