You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Divya Gehlot <di...@gmail.com> on 2016/02/15 05:40:01 UTC
which master option to view current running job in Spark UI
Hi,
I have Hortonworks 2.3.4 cluster on EC2 and Have spark jobs as scala files
.
I am bit confused between using *master *options
I want to execute this spark job in YARN
Curently running as
spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars
/usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path
/usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages
com.databricks:spark-csv_2.10:1.1.0 *--master yarn-client * -i
/TestDivya/Spark/Test.scala
with this option I cant see the currently running jobs in Spark WEB UI
though it later appear in spark history server.
My question with which --master option should I run my spark jobs so that I
can view the currently running jobs in spark web UI .
Thanks,
Divya
Re: which master option to view current running job in Spark UI
Posted by Divya Gehlot <di...@gmail.com>.
Hi Jeff ,
The issues with EC2 logs view .
Had to set up SSH tunnels to view the current running job.
Thanks,
Divya
On 24 February 2016 at 10:33, Jeff Zhang <zj...@gmail.com> wrote:
> View running job in SPARK UI doesn't matter which master you use. What do
> you mean "I cant see the currently running jobs in Spark WEB UI" ? Do you
> see a blank spark ui or can't open the spark ui ?
>
> On Mon, Feb 15, 2016 at 12:55 PM, Sabarish Sasidharan <
> sabarish.sasidharan@manthan.com> wrote:
>
>> When running in YARN, you can use the YARN Resource Manager UI to get to
>> the ApplicationMaster url, irrespective of client or cluster mode.
>>
>> Regards
>> Sab
>> On 15-Feb-2016 10:10 am, "Divya Gehlot" <di...@gmail.com> wrote:
>>
>>> Hi,
>>> I have Hortonworks 2.3.4 cluster on EC2 and Have spark jobs as scala
>>> files .
>>> I am bit confused between using *master *options
>>> I want to execute this spark job in YARN
>>>
>>> Curently running as
>>> spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars
>>> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path
>>> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages
>>> com.databricks:spark-csv_2.10:1.1.0 *--master yarn-client * -i
>>> /TestDivya/Spark/Test.scala
>>>
>>> with this option I cant see the currently running jobs in Spark WEB UI
>>> though it later appear in spark history server.
>>>
>>> My question with which --master option should I run my spark jobs so
>>> that I can view the currently running jobs in spark web UI .
>>>
>>> Thanks,
>>> Divya
>>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: which master option to view current running job in Spark UI
Posted by Jeff Zhang <zj...@gmail.com>.
View running job in SPARK UI doesn't matter which master you use. What do
you mean "I cant see the currently running jobs in Spark WEB UI" ? Do you
see a blank spark ui or can't open the spark ui ?
On Mon, Feb 15, 2016 at 12:55 PM, Sabarish Sasidharan <
sabarish.sasidharan@manthan.com> wrote:
> When running in YARN, you can use the YARN Resource Manager UI to get to
> the ApplicationMaster url, irrespective of client or cluster mode.
>
> Regards
> Sab
> On 15-Feb-2016 10:10 am, "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>> I have Hortonworks 2.3.4 cluster on EC2 and Have spark jobs as scala
>> files .
>> I am bit confused between using *master *options
>> I want to execute this spark job in YARN
>>
>> Curently running as
>> spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars
>> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path
>> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages
>> com.databricks:spark-csv_2.10:1.1.0 *--master yarn-client * -i
>> /TestDivya/Spark/Test.scala
>>
>> with this option I cant see the currently running jobs in Spark WEB UI
>> though it later appear in spark history server.
>>
>> My question with which --master option should I run my spark jobs so that
>> I can view the currently running jobs in spark web UI .
>>
>> Thanks,
>> Divya
>>
>
--
Best Regards
Jeff Zhang
Re: which master option to view current running job in Spark UI
Posted by Sabarish Sasidharan <sa...@manthan.com>.
When running in YARN, you can use the YARN Resource Manager UI to get to
the ApplicationMaster url, irrespective of client or cluster mode.
Regards
Sab
On 15-Feb-2016 10:10 am, "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
> I have Hortonworks 2.3.4 cluster on EC2 and Have spark jobs as scala files
> .
> I am bit confused between using *master *options
> I want to execute this spark job in YARN
>
> Curently running as
> spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars
> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path
> /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages
> com.databricks:spark-csv_2.10:1.1.0 *--master yarn-client * -i
> /TestDivya/Spark/Test.scala
>
> with this option I cant see the currently running jobs in Spark WEB UI
> though it later appear in spark history server.
>
> My question with which --master option should I run my spark jobs so that
> I can view the currently running jobs in spark web UI .
>
> Thanks,
> Divya
>