You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by paja <vy...@gmail.com> on 2015/01/07 18:27:54 UTC
Join stucks in the last stage step
Hello,
I have problem with join of two tables via Spark - I have tried to do it
via Spark SQL and API but no progress so far. I have basicaly two tables
ACCONTS - 16 mio records and TRANSACTIONS 2,5 billion records. When I try to
join the tables (please see code) the job stucks in the last stage for very
long (please see console output). And after eg 2h it writes to the output a
weird exception like
/org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0/
I have tried several strategies - repartitioning of RDDs, broadcast the
smaller one, but result is always same
Have sombody idea what happens?
Source Code. AccJoin.java
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n21018/AccJoin.java>
Console AccJoin_0.html
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n21018/AccJoin_0.html>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-stucks-in-the-last-stage-step-tp21018.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Join stucks in the last stage step
Posted by paja <vy...@gmail.com>.
Just to demonstrate BIG difference between ordinary task (id 450) and last
remaining task (id 0)
Index ID Attempt Status ▾ Locality Level Launch Time Duration GC Time
Shuffle Read Shuffle Spill (Memory) Shuffle Spill (Disk) Errors
0 2413 0 RUNNING 2015/01/08 08:03:19 *1.5 h 10.9 GB 176.9 GB* 4.4 GB
450 2863 0 SUCCESS 2015/01/08 08:06:55 *1.1 min 42.7 MB 968.7 MB* 32.9 MB
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-stucks-in-the-last-stage-step-tp21018p21028.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org