You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Grandl Robert <rg...@yahoo.com.INVALID> on 2015/03/12 16:33:16 UTC

run spark standalone mode

Hi guys,
I have a stupid question, but I am not sure how to get out of it. 
I deployed spark 1.2.1 on a cluster of 30 nodes. Looking at master:8088 I can see all the workers I have created so far. (I start the cluster with sbin/start-all.sh)
However, when running a Spark SQL query or even spark-shell, I cannot see any job executing at master webUI, but the jobs are able to finish. I suspect they are executing locally on the master, but I don't understand why/how and why not on slave machines. 

My conf/spark-env.sh is as following:export SPARK_MASTER_IP="ms0220"
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/users/rgrandl/software/spark-1.2.1-bin-hadoop2.4/lib/snappy-java-1.0.4.1.jar

export SPARK_LOCAL_DIRS="/users/rgrandl/software/data/spark/local"

export SPARK_WORKER_MEMORY="52000M"
export SPARK_WORKER_INSTANCES="2"
export SPARK_WORKER_CORES="2"

export SPARK_WORKER_DIR="/users/rgrandl/software/data/spark/worker"
export SPARK_DAEMON_MEMORY="5200M"
#export SPARK_DAEMON_JAVA_OPTS="4800M"


While conf/slaves is populated with the list of machines used for workers. I have to mention that spark-env.sh and slaves files are deployed on all machines. 

Thank you,Robert

Re: run spark standalone mode

Posted by Grandl Robert <rg...@yahoo.com.INVALID>.

Sorry guys for this.

It seems that I need to start the thrift server with --master spark://ms0220:7077 option and now I can see applications running in my web UI.
Thanks,Robert

On Thursday, March 12, 2015 10:57 AM, Grandl Robert <rg...@yahoo.com.INVALID> wrote:

I figured out for spark-shell by passing the --master option. However, still troubleshooting for launching sql queries. My current command is like that:
./bin/beeline -u jdbc:hive2://ms0220:10000 -n `whoami` -p ignored -f tpch_query10.sql

On Thursday, March 12, 2015 10:37 AM, Grandl Robert <rg...@yahoo.com.INVALID> wrote:

Hi guys,
I have a stupid question, but I am not sure how to get out of it.
I deployed spark 1.2.1 on a cluster of 30 nodes. Looking at master:8088 I can see all the workers I have created so far. (I start the cluster with sbin/start-all.sh)
However, when running a Spark SQL query or even spark-shell, I cannot see any job executing at master webUI, but the jobs are able to finish. I suspect they are executing locally on the master, but I don't understand why/how and why not on slave machines.

My conf/spark-env.sh is as following:export SPARK_MASTER_IP="ms0220"
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/users/rgrandl/software/spark-1.2.1-bin-hadoop2.4/lib/snappy-java-1.0.4.1.jar

export SPARK_LOCAL_DIRS="/users/rgrandl/software/data/spark/local"

export SPARK_WORKER_MEMORY="52000M"
export SPARK_WORKER_INSTANCES="2"
export SPARK_WORKER_CORES="2"

export SPARK_WORKER_DIR="/users/rgrandl/software/data/spark/worker"
export SPARK_DAEMON_MEMORY="5200M"
#export SPARK_DAEMON_JAVA_OPTS="4800M"

While conf/slaves is populated with the list of machines used for workers. I have to mention that spark-env.sh and slaves files are deployed on all machines.

Thank you,Robert