You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by wxhsdp <wx...@gmail.com> on 2014/04/23 05:21:49 UTC

no response in spark web UI

Hi, all
  i used to run my app using sbt run. but now i want to see the job
information in spark web ui. 
i'am in local mode, i start the spark shell, and access the web ui on
http://ubuntu.local:4040/stages/.
but when i sbt run some application, there is no response in the web ui.

how to make connection between sbt and the web ui?
appreciate your help!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/no-response-in-spark-web-UI-tp4633.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: no response in spark web UI

Posted by wxhsdp <wx...@gmail.com>.

thanks for help
now i'am testing in standalone mode in only one pc. 
i use ./sbin/start-master.sh to start a master and 
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu:7077
to connect to the master 

from the web ui, i can see the local worker registered with 2.9GB memory 
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n4650/Screenshot_from_2014-04-23_16%5E%2551%5E%2553.png> 

but when i run whatever applications(for example the SimpleApp.scala in
quick start with input file=4.5KB), it failed with 
14/04/23 16:39:13 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0) 
14/04/23 16:39:13 WARN scheduler.TaskSetManager: Loss was due to
java.lang.OutOfMemoryError 
java.lang.OutOfMemoryError: Java heap space 
        at
org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183) 
        at
org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2378) 
        at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285) 
        at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77) 
        at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:606) 
        at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) 
        at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) 
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) 
        at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40) 
        at
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:165) 
        at
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:606) 
        at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) 
        at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) 
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
        at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 
        at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
        at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 
        at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 

i use sbt to run the application 
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=256M" 
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@" 

what's the problem?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/no-response-in-spark-web-UI-tp4633p4650.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: no response in spark web UI

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Hi

SparkContext launches the web interface at 4040, if you have multiple
sparkContext's on the same machine then the ports will be bind to
successive ports beginning with 4040.

Here's the documentation:
https://spark.apache.org/docs/0.9.0/monitoring.html

And here's a simple scala program to starts with :
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala


Thanks
Best Regards.


On Wed, Apr 23, 2014 at 8:51 AM, wxhsdp <wx...@gmail.com> wrote:

> Hi, all
>   i used to run my app using sbt run. but now i want to see the job
> information in spark web ui.
> i'am in local mode, i start the spark shell, and access the web ui on
> http://ubuntu.local:4040/stages/.
> but when i sbt run some application, there is no response in the web ui.
>
> how to make connection between sbt and the web ui?
> appreciate your help!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/no-response-in-spark-web-UI-tp4633.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>