Spark jobs run extremely slow on yarn cluster compared to standalone spark

[pdesai <pd...@cloudfabrix.com>]

Hi there,

I am doing a POC with Spark and I have noticed that if I run my job on
standalone spark installation, it finishes in a second(It's a small sample
job). But when I run same job on spark cluster with Yarn, it takes 4-5 min
in simple execution. 
Are there any best practices that I need to follow for spark cluster
configuration. I have left all default settings. During spark-submit I
specify num-executors=3, executor-memory=512m, executor-cores-1.

I am using Java Spark SQL API.

Thanks,
Purvi



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-jobs-run-extremely-slow-on-yarn-cluster-compared-to-standalone-spark-tp26215.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark jobs run extremely slow on yarn cluster compared to standalone spark

["Yuval.Itzchakov" <yu...@gmail.com>]

Your question lacks sufficient information for us to actually provide help.
Have you looked at the Spark UI to see which part of the graph is taking the
longest? Have you tried logging your methods?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-jobs-run-extremely-slow-on-yarn-cluster-compared-to-standalone-spark-tp26215p26221.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org