You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by David Thomas <dt...@gmail.com> on 2014/02/19 04:06:29 UTC
Question on web UI
On the spark web UI, there are two tables - Running Applications and
Running Drivers. What is the difference between these two?
When I submit an application to run on the cluster, I can see the driver
table getting populated, but why is only one worker listed under Worker
column? What is the role of this single worker? Shouldn't the driver
program be running on the master node?
Re: Question on web UI
Posted by Nan Zhu <zh...@gmail.com>.
you can check from here http://spark.incubator.apache.org/docs/latest/cluster-overview.html
the driver is more like a per-application task scheduler whereas the master is a resource manager to allocate resources to applications and maintain workers
with the command you use, the driver will run on the worker nodes
Best,
--
Nan Zhu
On Tuesday, February 18, 2014 at 10:32 PM, David Thomas wrote:
> I was using that exact command line (./bin/spark-class org.apache.spark.deploy.Client launch) to launch the app within the cluster. Now I was running this from the master node, but the Worker column shows just one worker in the cluster. So what is the significance of that column? Shouldn't the driver be running on the master node?
>
>
> On Tue, Feb 18, 2014 at 8:31 PM, Nan Zhu <zhunanmcgill@gmail.com (mailto:zhunanmcgill@gmail.com)> wrote:
> > the driver is running on the machine where you run command like ./spark-shell
> >
> > but in 0.9, you can run in-cluster driver http://spark.incubator.apache.org/docs/latest/spark-standalone.html#launching-applications-inside-the-cluster
> >
> > Best,
> >
> > --
> > Nan Zhu
> >
> >
> > On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:
> >
> > > On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?
> > >
> > > When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?
> >
>
Re: Question on web UI
Posted by David Thomas <dt...@gmail.com>.
I was using that exact command line (./bin/spark-class
org.apache.spark.deploy.Client launch) to launch the app within the
cluster. Now I was running this from the master node, but the Worker column
shows just one worker in the cluster. So what is the significance of that
column? Shouldn't the driver be running on the master node?
On Tue, Feb 18, 2014 at 8:31 PM, Nan Zhu <zh...@gmail.com> wrote:
> the driver is running on the machine where you run command like
> ./spark-shell
>
> but in 0.9, you can run in-cluster driver
> http://spark.incubator.apache.org/docs/latest/spark-standalone.html#launching-applications-inside-the-cluster
>
> Best,
>
> --
> Nan Zhu
>
> On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:
>
> On the spark web UI, there are two tables - Running Applications and
> Running Drivers. What is the difference between these two?
>
> When I submit an application to run on the cluster, I can see the driver
> table getting populated, but why is only one worker listed under Worker
> column? What is the role of this single worker? Shouldn't the driver
> program be running on the master node?
>
>
>
Re: Question on web UI
Posted by Nan Zhu <zh...@gmail.com>.
the driver is running on the machine where you run command like ./spark-shell
but in 0.9, you can run in-cluster driver http://spark.incubator.apache.org/docs/latest/spark-standalone.html#launching-applications-inside-the-cluster
Best,
--
Nan Zhu
On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:
> On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?
>
> When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?