GraphX can not unpersist edges of old graph?

["余根茂(余根茂)" <ge...@alibaba-inc.com>]

Hi All,

         

I know the benefit of RDD caching, but abuse of using cache may cause memory
leak. In graphx, we can cache a graph by using graph.cache(), and many
transformation of graph create and cache new edges, like partitionBy() and
subgraph(). However, I can not find an interface to unpersist edges. I
wonder the purpose of design like this. It can indeed improve performance.
But, it may lead to memory leak if not unpersist edges in some cases. For
example, in Spark-1.0,  the pregel can not unpersist edges of old graph
effectively, and leads to memory leak.

 

 

Much Thanks!

 

Re: GraphX can not unpersist edges of old graph?

[Ankur Dave <an...@gmail.com>]

We didn't provide an unpersist API for Graph because the internal
dependency structure of a graph can make it hard to unpersist correctly in
a way that avoids recomputation. However, you can directly unpersist a
graph's vertices and edges RDDs using graph.vertices.unpersist() and
graph.edges.unpersist().

By the way, the memory leak bug with Pregel (SPARK-2025
<https://issues.apache.org/jira/browse/SPARK-2025>) is fixed in master.

Ankur <http://www.ankurdave.com/>