You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by MEETHU MATHEW <me...@yahoo.co.in> on 2014/07/17 10:05:28 UTC

Pysparkshell are not listing in the web UI while running


 Hi all,

I just upgraded to spark 1.0.1. In spark 1.0.0 when I start Ipython notebook using the following command,it used to come in the running applications tab in master:8080 web UI.

IPYTHON_OPTS="notebook --pylab inline" $SPARK_HOME/bin/pyspark

But now when I run it,its not getting listed under running application/completed application(once its closed).But I am able to see the spark stages at master:4040 while its running

Anyone have any idea why this 



Thanks & Regards, 
Meethu M

Re: Pysparkshell are not listing in the web UI while running

Posted by MEETHU MATHEW <me...@yahoo.co.in>.
Hi Akhil,

That fixed the problem...Thanks

 
Thanks & Regards, 
Meethu M


On Thursday, 17 July 2014 2:26 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
 


Hi Neethu,

Your application is running on local mode and that's the reason why you are not seeing the driver app in the 8080 webUI. You can pass the Master IP to your pyspark and get it running in cluster mode. 

eg: IPYTHON_OPTS="notebook --pylab inline" $SPARK_HOME/bin/pyspark --master spark://master:7077


Replace master:7077 with the spark uri that you are seeing in top left of the 8080 webui.



Thanks
Best Regards


On Thu, Jul 17, 2014 at 1:35 PM, MEETHU MATHEW <me...@yahoo.co.in> wrote:


>
> Hi all,
>
>
>I just upgraded to spark 1.0.1. In spark 1.0.0 when I start Ipython notebook using the following command,it used to come in the running applications tab in master:8080 web UI.
>
>
>IPYTHON_OPTS="notebook --pylab inline" $SPARK_HOME/bin/pyspark
>
>
>But now when I run it,its not getting listed under running application/completed application(once its closed).But I am able to see the spark stages at master:4040 while its running
>
>
>Anyone have any idea why this 
>
>
>
>
>
>
>Thanks & Regards, 
>Meethu M

Re: Pysparkshell are not listing in the web UI while running

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Hi Neethu,

Your application is running on local mode and that's the reason why you are
not seeing the driver app in the 8080 webUI. You can pass the Master IP to
your pyspark and get it running in cluster mode.

eg: IPYTHON_OPTS="notebook --pylab inline" $SPARK_HOME/bin/pyspark --master
spark://master:7077


Replace master:7077 with the spark uri that you are seeing in top left of
the 8080 webui.


Thanks
Best Regards


On Thu, Jul 17, 2014 at 1:35 PM, MEETHU MATHEW <me...@yahoo.co.in>
wrote:

>
>  Hi all,
>
> I just upgraded to spark 1.0.1. In spark 1.0.0 when I start Ipython
> notebook using the following command,it used to come in the running
> applications tab in master:8080 web UI.
>
> IPYTHON_OPTS="notebook --pylab inline" $SPARK_HOME/bin/pyspark
>
> But now when I run it,its not getting listed under running
> application/completed application(once its closed).But I am able to see the
> spark stages at master:4040 while its running
>
> Anyone have any idea why this
>
>
>
> Thanks & Regards,
> Meethu M
>