You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2017/05/19 19:05:01 UTC

Shuffle read is very slow in spark

Hi ,

I am in weird situation where the spark job  behaves abnormal. sometimes it
runs very fast and some times it takes long time to complete. Not sure what
is going on. Cluster is free most of the times.

Below image shows the shuffle read is still taking more than 3 hours to
write data back into hive table using sql.context("Insert overwrirte table
hivetable from select * from spark_temporary_table")

[image: Inline image 1]

could anyone let me know how to resolve this and run the job faster.

Thanks,
Asmath