Failed fetch: "Could not get block(s)"

[Al M <al...@gmail.com>]

I am using Spark 1.1.1.  I am seeing an issue that only appears when I run in
standalone clustered mode with at least 2 workers.  The workers are on
separate physical machines.
I am performing a simple join on 2 RDDs.  After the join I run first() on
the joined RDD (in Scala) to get the first result.  When this first() runs
on Worker A it works fine; when the first() runs on worker B I get an error
'Fetch Failure'.
I looked at the work stderr log for worker B.  It shows the following
exception:
INFO BlockFetcherInterator$BasicBlockFetcherIterator: Started 2 remote
fetches in 2 msERROR BlockFetcherIterator$BasicBlockFetcherIterator: Could
not get block(s) from ConnectionManagerId(, )java.io.IOException:
sendMessageReliably failed because ack was not received within 60 sec    at
org.apache.spark.network.ConnectionManager$$anon$10$$anonfun$run$15.apply(ConnectionManager.scala:866).....
It is trying to connect to the ConnectionManager for BlockManager on Worker
A from Worker B.  It manages to connect, but it always times out.  When I
try to connect via telnet I see the same: it connects, but I don't get
anything back from the host
I noticed that two other people reported  this issue
<http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-Freezing-while-running-TPC-H-query-5-td14902.html> 
.  Unfortunately there was no meaningful progress.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Failed-fetch-Could-not-get-block-s-tp20262.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.