Re: Spark 2.1.1 Graphx graph loader GC overhead error

[Aritra Mandal <ar...@gmail.com>]

yncxcw wrote
> hi,
> 
> It highly depends on the algorithms you are going to apply to your data
> sets.  Graph applications are usually memory hungry and probably cause
> long
> GC or even OOM.
> 
> Suggestions include:  1. make some highly reused RDD as
> StorageLevel.MEMORY_ONLY
> and leave the rest  MEMORY_AND_DISK.
> 
>                                     2. slight decrease the parallelism for
> each executor.
> 
> 
> Wei Chen


Thanks for the response have a implementation of K core decomposition
running using pregel framework.

I will try constructing the graph with storagelevel:MEMORY_AND_DISK and 
post the outcome here

The GC overhead error is happening even before the algorithm starts its
pregel iterations it failing in the GraphLoader.fromEdgeList stage.

Aritra



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-1-1-Graphx-graph-loader-GC-overhead-error-tp28841p28843.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Spark 2.1.1 Graphx graph loader GC overhead error

[Aritra Mandal <ar...@gmail.com>]

yncxcw wrote
> hi,
> 
> I think if the OOM occurs before the computation begins, the input data is
> probably too big to fit in memory. I remembered that the graph data would
> expand when loading the data input memory. And the scale of expanding is
> pretty huge( based on my experiment on Pagerank).
> 
> 
> Wei  Chen

Hello Wei,

Thanks for the suggestions.

I tried this small piece of code with StorageLevel.MEMORY_AND_DISK I removed
the pregel call just to test.
But still the code failed with OOM in the graphload stage

/val ygraph=GraphLoader.edgeListFile(sc,args(1),
true,32,StorageLevel.MEMORY_AND_DISK,StorageLevel.MEMORY_AND_DISK).partitionBy(PartitionStrategy.RandomVertexCut)

println(ygraph.vertices.count())/


Is there a way to calculate the maximum size of a graph that a given
configuration of the cluster can process.

Aritra





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-1-1-Graphx-graph-loader-GC-overhead-error-tp28841p28851.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org