You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by "Chawla,Sumit " <su...@gmail.com> on 2016/12/10 05:42:56 UTC

Spark Mesos Architecture Details and Integration details

Hi All

I am a new member of this group. Sorry if this question has already been
asked.  I am looking for some information regarding Spark Mesos Integration:

1.  How does Mesos schedules and launch Spark Executors? (Any pointer to
code will be helpful)

2.  How does Mesos frontend the Spark Administration capabilities?  Are
there any urls through which mesos proxy's the request for spark master.  I
am particularly interested to find out how to access Spark Master Admin and
Monitoring URLs when running using Mesos.

3.  Any other good documentations/architecture for me to get started on
Mesos and Spark integration and internals.


Regards
Sumit Chawla

Re: Spark Mesos Architecture Details and Integration details

Posted by Michael Gummelt <mg...@mesosphere.io>.
On Fri, Dec 9, 2016 at 9:42 PM, Chawla,Sumit <su...@gmail.com> wrote:

> Hi All
>
> I am a new member of this group. Sorry if this question has already been
> asked.  I am looking for some information regarding Spark Mesos Integration:
>
> 1.  How does Mesos schedules and launch Spark Executors? (Any pointer to
> code will be helpful)
>

This is the scheduler:
https://github.com/apache/spark/blob/master/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L48


>
> 2.  How does Mesos frontend the Spark Administration capabilities?  Are
> there any urls through which mesos proxy's the request for spark master.  I
> am particularly interested to find out how to access Spark Master Admin and
> Monitoring URLs when running using Mesos.
>

The driver's UI can be viewed on the node running the driver, by default on
port 4040.  Since this dies when the driver dies, you probably want to set
up the history server http://spark.apache.org/docs/latest/monitoring.html


>
> 3.  Any other good documentations/architecture for me to get started on
> Mesos and Spark integration and internals.
>

Just the docs, and maybe read through the JIRAs and old emails.


>
>
> Regards
> Sumit Chawla
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere