You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hellen <ho...@gmail.com> on 2015/07/05 00:57:44 UTC
Spark got stuck with BlockManager after computing connected
components using GraphX
I'm computing connected components using Spark GraphX on AWS EC2. I believe
the computation was successful, as I saw the type information of the final
result. However, it looks like Spark was doing some cleanup. The
BlockManager removed a bunch of blocks and stuck at
15/07/04 21:53:06 INFO storage.BlockManager: Removing block rdd_334_4
15/07/04 21:53:06 INFO storage.MemoryStore: Block rdd_334_4 of size 25986936
dropped from memory (free 15648106262)
There was no error message, no update for like an hour. If I press the Enter
key, I got disconnected from the cluster. Does anyone happen to know what's
going on here?
I used 8 r3.4xlarge instances. I have 7 million edges and 200 million
vertices.
Thank you!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-got-stuck-with-BlockManager-after-computing-connected-components-using-GraphX-tp23620.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark got stuck with BlockManager after computing connected
components using GraphX
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
If you don't want those logs flood your screen, you can disable it simply
with:
import org.apache.log4j.{Level, Logger}
> Logger.getLogger("org").setLevel(Level.OFF)
> Logger.getLogger("akka").setLevel(Level.OFF)
Thanks
Best Regards
On Sun, Jul 5, 2015 at 7:27 PM, Hellen <ho...@gmail.com> wrote:
> Sorry for the silly question. I'm fairly new to Spark.
> Because of the cleanup log messages, I didn't see "scala>", so I thought
> it's still working on something. If I press "Enter", I got disconnected. I
> finally tried typing the variable name, which actually worked.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-got-stuck-with-BlockManager-after-computing-connected-components-using-GraphX-tp23620p23623.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark got stuck with BlockManager after computing connected
components using GraphX
Posted by Hellen <ho...@gmail.com>.
Sorry for the silly question. I'm fairly new to Spark.
Because of the cleanup log messages, I didn't see "scala>", so I thought
it's still working on something. If I press "Enter", I got disconnected. I
finally tried typing the variable name, which actually worked.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-got-stuck-with-BlockManager-after-computing-connected-components-using-GraphX-tp23620p23623.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org