You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anupama Joshi <an...@gmail.com> on 2015/03/03 00:39:30 UTC
Spark UI and running spark-submit with --master yarn
Hi ,
1. When I run my application with --master yarn-cluster or --master
yarn --deploy-mode cluster , I can not the spark UI at the location --
<masternode>:4040Even if I am running the job , I can not see teh SPARK UI.
2. When I run with --master yarn --deploy-mode client -- I see the
Spark UI but I cannot see my job running.
When I run spark-submit with --master local[*] , I see the spark UI , my
job everything (Thats great)
Do I need to do some settings to see the UI?
Thanks
-AJ
Re: Spark UI and running spark-submit with --master yarn
Posted by Marcelo Vanzin <va...@cloudera.com>.
That does not look like the RM UI. Please check your configuration for
the port (see Ted's e-mail).
On Mon, Mar 2, 2015 at 4:45 PM, Anupama Joshi <an...@gmail.com> wrote:
> Hi ,
> port 8088 does not show me anything .(can not connect)
> where as port ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9026 shows
> me all the applications.
> Do I have to do anything for the port 8088 or whatever I am seeing at 9026
> port is good .Attached is screenshot .
> Thanks
> AJ
>
> On Mon, Mar 2, 2015 at 4:24 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> That's the RM's RPC port, not the web UI port. (See Ted's e-mail -
>> normally web UI is on 8088.)
>>
>> On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi <an...@gmail.com>
>> wrote:
>> > Hi Marcelo,
>> > Thanks for the quick reply.
>> > I have a EMR cluster and I am running the spark-submit on the master
>> > node in
>> > the cluster.
>> > When I start the spark-submit , I see
>> > 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at
>> > /172.31.43.254:9022
>> > But If I try that URL or the use the external DNS
>> > ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
>> > it does not work
>> > What am I missing here ?
>> > Thanks a lot for the help
>> > -AJ
>> >
>> >
>> > On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <va...@cloudera.com>
>> > wrote:
>> >>
>> >> What are you calling "<masternode>"? In yarn-cluster mode, the driver
>> >> is running somewhere in your cluster, not on the machine where you run
>> >> spark-submit.
>> >>
>> >> The easiest way to get to the Spark UI when using Yarn is to use the
>> >> Yarn RM's web UI. That will give you a link to the application's UI
>> >> regardless of whether it's running on client or cluster mode.
>> >>
>> >> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com>
>> >> wrote:
>> >> > Hi ,
>> >> >
>> >> > When I run my application with --master yarn-cluster or --master
>> >> > yarn
>> >> > --deploy-mode cluster , I can not the spark UI at the location --
>> >> > <masternode>:4040Even if I am running the job , I can not see teh
>> >> > SPARK
>> >> > UI.
>> >> > When I run with --master yarn --deploy-mode client -- I see the
>> >> > Spark
>> >> > UI
>> >> > but I cannot see my job running.
>> >> >
>> >> > When I run spark-submit with --master local[*] , I see the spark UI ,
>> >> > my
>> >> > job
>> >> > everything (Thats great)
>> >> >
>> >> > Do I need to do some settings to see the UI?
>> >> >
>> >> > Thanks
>> >> >
>> >> > -AJ
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Marcelo
>> >
>> >
>>
>>
>>
>> --
>> Marcelo
>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark UI and running spark-submit with --master yarn
Posted by Anupama Joshi <an...@gmail.com>.
Hi ,
port 8088 does not show me anything .(can not connect)
where as port ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9026 shows
me all the applications.
Do I have to do anything for the port 8088 or whatever I am seeing at 9026
port is good .Attached is screenshot .
Thanks
AJ
On Mon, Mar 2, 2015 at 4:24 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> That's the RM's RPC port, not the web UI port. (See Ted's e-mail -
> normally web UI is on 8088.)
>
> On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi <an...@gmail.com>
> wrote:
> > Hi Marcelo,
> > Thanks for the quick reply.
> > I have a EMR cluster and I am running the spark-submit on the master
> node in
> > the cluster.
> > When I start the spark-submit , I see
> > 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at
> > /172.31.43.254:9022
> > But If I try that URL or the use the external DNS
> > ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
> > it does not work
> > What am I missing here ?
> > Thanks a lot for the help
> > -AJ
> >
> >
> > On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
> >>
> >> What are you calling "<masternode>"? In yarn-cluster mode, the driver
> >> is running somewhere in your cluster, not on the machine where you run
> >> spark-submit.
> >>
> >> The easiest way to get to the Spark UI when using Yarn is to use the
> >> Yarn RM's web UI. That will give you a link to the application's UI
> >> regardless of whether it's running on client or cluster mode.
> >>
> >> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com>
> >> wrote:
> >> > Hi ,
> >> >
> >> > When I run my application with --master yarn-cluster or --master yarn
> >> > --deploy-mode cluster , I can not the spark UI at the location --
> >> > <masternode>:4040Even if I am running the job , I can not see teh
> SPARK
> >> > UI.
> >> > When I run with --master yarn --deploy-mode client -- I see the Spark
> >> > UI
> >> > but I cannot see my job running.
> >> >
> >> > When I run spark-submit with --master local[*] , I see the spark UI ,
> my
> >> > job
> >> > everything (Thats great)
> >> >
> >> > Do I need to do some settings to see the UI?
> >> >
> >> > Thanks
> >> >
> >> > -AJ
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Marcelo
> >
> >
>
>
>
> --
> Marcelo
>
Re: Spark UI and running spark-submit with --master yarn
Posted by Marcelo Vanzin <va...@cloudera.com>.
That's the RM's RPC port, not the web UI port. (See Ted's e-mail -
normally web UI is on 8088.)
On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi <an...@gmail.com> wrote:
> Hi Marcelo,
> Thanks for the quick reply.
> I have a EMR cluster and I am running the spark-submit on the master node in
> the cluster.
> When I start the spark-submit , I see
> 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at
> /172.31.43.254:9022
> But If I try that URL or the use the external DNS
> ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
> it does not work
> What am I missing here ?
> Thanks a lot for the help
> -AJ
>
>
> On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> What are you calling "<masternode>"? In yarn-cluster mode, the driver
>> is running somewhere in your cluster, not on the machine where you run
>> spark-submit.
>>
>> The easiest way to get to the Spark UI when using Yarn is to use the
>> Yarn RM's web UI. That will give you a link to the application's UI
>> regardless of whether it's running on client or cluster mode.
>>
>> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com>
>> wrote:
>> > Hi ,
>> >
>> > When I run my application with --master yarn-cluster or --master yarn
>> > --deploy-mode cluster , I can not the spark UI at the location --
>> > <masternode>:4040Even if I am running the job , I can not see teh SPARK
>> > UI.
>> > When I run with --master yarn --deploy-mode client -- I see the Spark
>> > UI
>> > but I cannot see my job running.
>> >
>> > When I run spark-submit with --master local[*] , I see the spark UI , my
>> > job
>> > everything (Thats great)
>> >
>> > Do I need to do some settings to see the UI?
>> >
>> > Thanks
>> >
>> > -AJ
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Marcelo
>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark UI and running spark-submit with --master yarn
Posted by Ted Yu <yu...@gmail.com>.
Default RM Web UI port is 8088 (configurable
through yarn.resourcemanager.webapp.address)
Cheers
On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi <an...@gmail.com>
wrote:
> Hi Marcelo,
> Thanks for the quick reply.
> I have a EMR cluster and I am running the spark-submit on the master node
> in the cluster.
> When I start the spark-submit , I see
> 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.43.254:9022
> But If I try that URL or the use the external DNS
> ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
> it does not work
> What am I missing here ?
> Thanks a lot for the help
> -AJ
>
>
> On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> What are you calling "<masternode>"? In yarn-cluster mode, the driver
>> is running somewhere in your cluster, not on the machine where you run
>> spark-submit.
>>
>> The easiest way to get to the Spark UI when using Yarn is to use the
>> Yarn RM's web UI. That will give you a link to the application's UI
>> regardless of whether it's running on client or cluster mode.
>>
>> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com>
>> wrote:
>> > Hi ,
>> >
>> > When I run my application with --master yarn-cluster or --master yarn
>> > --deploy-mode cluster , I can not the spark UI at the location --
>> > <masternode>:4040Even if I am running the job , I can not see teh SPARK
>> UI.
>> > When I run with --master yarn --deploy-mode client -- I see the Spark
>> UI
>> > but I cannot see my job running.
>> >
>> > When I run spark-submit with --master local[*] , I see the spark UI ,
>> my job
>> > everything (Thats great)
>> >
>> > Do I need to do some settings to see the UI?
>> >
>> > Thanks
>> >
>> > -AJ
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>
>
Re: Spark UI and running spark-submit with --master yarn
Posted by Anupama Joshi <an...@gmail.com>.
Hi Marcelo,
Thanks for the quick reply.
I have a EMR cluster and I am running the spark-submit on the master node
in the cluster.
When I start the spark-submit , I see
15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at /
172.31.43.254:9022
But If I try that URL or the use the external DNS
ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
it does not work
What am I missing here ?
Thanks a lot for the help
-AJ
On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> What are you calling "<masternode>"? In yarn-cluster mode, the driver
> is running somewhere in your cluster, not on the machine where you run
> spark-submit.
>
> The easiest way to get to the Spark UI when using Yarn is to use the
> Yarn RM's web UI. That will give you a link to the application's UI
> regardless of whether it's running on client or cluster mode.
>
> On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com>
> wrote:
> > Hi ,
> >
> > When I run my application with --master yarn-cluster or --master yarn
> > --deploy-mode cluster , I can not the spark UI at the location --
> > <masternode>:4040Even if I am running the job , I can not see teh SPARK
> UI.
> > When I run with --master yarn --deploy-mode client -- I see the Spark UI
> > but I cannot see my job running.
> >
> > When I run spark-submit with --master local[*] , I see the spark UI , my
> job
> > everything (Thats great)
> >
> > Do I need to do some settings to see the UI?
> >
> > Thanks
> >
> > -AJ
> >
> >
> >
> >
> >
> >
>
>
>
> --
> Marcelo
>
Re: Spark UI and running spark-submit with --master yarn
Posted by Marcelo Vanzin <va...@cloudera.com>.
What are you calling "<masternode>"? In yarn-cluster mode, the driver
is running somewhere in your cluster, not on the machine where you run
spark-submit.
The easiest way to get to the Spark UI when using Yarn is to use the
Yarn RM's web UI. That will give you a link to the application's UI
regardless of whether it's running on client or cluster mode.
On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi <an...@gmail.com> wrote:
> Hi ,
>
> When I run my application with --master yarn-cluster or --master yarn
> --deploy-mode cluster , I can not the spark UI at the location --
> <masternode>:4040Even if I am running the job , I can not see teh SPARK UI.
> When I run with --master yarn --deploy-mode client -- I see the Spark UI
> but I cannot see my job running.
>
> When I run spark-submit with --master local[*] , I see the spark UI , my job
> everything (Thats great)
>
> Do I need to do some settings to see the UI?
>
> Thanks
>
> -AJ
>
>
>
>
>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org