You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arpit Kumar <ar...@gmail.com> on 2014/10/25 05:26:34 UTC
How to set persistence level of graph in GraphX in spark 1.0.0
Hi all,
I am using the GrpahLoader class to load graphs from edge list files. But
then I need to change the storage level of the graph to some other thing
than MEMORY_ONLY.
val graph = GraphLoader.edgeListFile(sc, fname,
minEdgePartitions =
numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER)
The error I am getting while executing this is:
Exception in thread "main" java.lang.UnsupportedOperationException: Cannot
change storage level of an RDD after it was already assigned a level
Then I looked into the GraphLoader class. I know that in the latest version
of spark support for setting persistence level is provided in this class.
Please suggest a workaround for spark 1.0.0 as I do not have the option to
shift to latest release.
Note: I tried copying the GraphLoader class to my package as GraphLoader1
importing
package com.cloudera.xyz
import org.apache.spark.storage.StorageLevel
import org.apache.spark.graphx._
import org.apache.spark.{Logging, SparkContext}
import org.apache.spark.graphx.impl._
and then changing the persistence level to my suitability as
.persist(gStorageLevel) instead of .cache()
But while compiling I am getting the following errors
GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl
cannot be accessed in package org.apache.spark.graphx.impl
[INFO] val builder = new EdgePartitionBuilder[Int, Int]
I am also attaching the file with the mail. Maybe this way of doing thing
is not possible.
Please suggest some workarounds so that I can set persistence level of my
graph to MEMORY_AND_DISK_SER for the graph I read from edge file list
Re: How to set persistence level of graph in GraphX in spark 1.0.0
Posted by Ankur Dave <an...@gmail.com>.
At 2014-10-25 08:56:34 +0530, Arpit Kumar <ar...@gmail.com> wrote:
> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl
> cannot be accessed in package org.apache.spark.graphx.impl
> [INFO] val builder = new EdgePartitionBuilder[Int, Int]
Here's a workaround:
1. Copy and modify the GraphLoader source as you did, but keep it in the org.apache.spark.graphx.impl package to fix the package-private error.
2. In addition to changing the persistence level of the edges RDD in GraphLoader, construct the VertexRDD and EdgeRDD yourself.
3. Call GraphImpl.fromExistingRDDs to construct the graph. This function will respect the existing EdgeRDD storage level.
4. Use the graph as desired. Be sure to avoid Graph#partitionBy, the Pregel API, and all of the built-in algorithms, because they call Graph#cache() on intermediate graphs.
Here is a modified version of GraphLoader that does 1-3: https://gist.github.com/ankurdave/0394d47809297eea76ff
Ankur
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to set persistence level of graph in GraphX in spark 1.0.0
Posted by Yifan LI <ia...@gmail.com>.
I am not sure if it can work on Spark 1.0, but give it a try.
or, Maybe you can try:
1) to construct the edges and vertices RDDs respectively with desired storage level.
2) then, to obtain a graph by using Graph(verticesRDD, edgesRDD).
Best,
Yifan LI
On 28 Oct 2014, at 12:10, Arpit Kumar <ar...@gmail.com> wrote:
> Hi Yifan LI,
> I am currently working on Spark 1.0 in which we can't pass edgeStorageLevel as parameter. It implicitly caches the edges. So I am looking for a workaround.
>
> http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.graphx.GraphLoader$
>
> Regards,
> Arpit
>
> On Tue, Oct 28, 2014 at 4:25 PM, Yifan LI <ia...@gmail.com> wrote:
> Hi Arpit,
>
> To try this:
>
> val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK,
> vertexStorageLevel = StorageLevel.MEMORY_AND_DISK)
>
>
> Best,
> Yifan LI
>
>
>
>
> On 28 Oct 2014, at 11:17, Arpit Kumar <ar...@gmail.com> wrote:
>
>> Any help regarding this issue please?
>>
>> Regards,
>> Arpit
>>
>> On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <ar...@gmail.com> wrote:
>> Hi all,
>> I am using the GrpahLoader class to load graphs from edge list files. But then I need to change the storage level of the graph to some other thing than MEMORY_ONLY.
>>
>> val graph = GraphLoader.edgeListFile(sc, fname,
>> minEdgePartitions = numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER)
>>
>> The error I am getting while executing this is:
>> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot change storage level of an RDD after it was already assigned a level
>>
>>
>> Then I looked into the GraphLoader class. I know that in the latest version of spark support for setting persistence level is provided in this class. Please suggest a workaround for spark 1.0.0 as I do not have the option to shift to latest release.
>>
>> Note: I tried copying the GraphLoader class to my package as GraphLoader1 importing
>>
>> package com.cloudera.xyz
>>
>> import org.apache.spark.storage.StorageLevel
>> import org.apache.spark.graphx._
>> import org.apache.spark.{Logging, SparkContext}
>> import org.apache.spark.graphx.impl._
>>
>> and then changing the persistence level to my suitability as
>> .persist(gStorageLevel) instead of .cache()
>>
>> But while compiling I am getting the following errors
>>
>> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl cannot be accessed in package org.apache.spark.graphx.impl
>> [INFO] val builder = new EdgePartitionBuilder[Int, Int]
>>
>> I am also attaching the file with the mail. Maybe this way of doing thing is not possible.
>>
>>
>> Please suggest some workarounds so that I can set persistence level of my graph to MEMORY_AND_DISK_SER for the graph I read from edge file list
>>
>>
>>
>> --
>> Arpit Kumar
>> Fourth Year Undergraduate
>> Department of Computer Science and Engineering
>> Indian Institute of Technology, Kharagpur
>
>
>
>
> --
> Arpit Kumar
> Fourth Year Undergraduate
> Department of Computer Science and Engineering
> Indian Institute of Technology, Kharagpur
Re: How to set persistence level of graph in GraphX in spark 1.0.0
Posted by Arpit Kumar <ar...@gmail.com>.
Hi Yifan LI,
I am currently working on Spark 1.0 in which we can't pass edgeStorageLevel
as parameter. It implicitly caches the edges. So I am looking for a
workaround.
http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.graphx.GraphLoader$
Regards,
Arpit
On Tue, Oct 28, 2014 at 4:25 PM, Yifan LI <ia...@gmail.com> wrote:
> Hi Arpit,
>
> To try this:
>
> val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions =
> numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK,
> vertexStorageLevel = StorageLevel.MEMORY_AND_DISK)
>
>
> Best,
> Yifan LI
>
>
>
>
> On 28 Oct 2014, at 11:17, Arpit Kumar <ar...@gmail.com> wrote:
>
> Any help regarding this issue please?
>
> Regards,
> Arpit
>
> On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <ar...@gmail.com> wrote:
>
>> Hi all,
>> I am using the GrpahLoader class to load graphs from edge list files. But
>> then I need to change the storage level of the graph to some other thing
>> than MEMORY_ONLY.
>>
>> val graph = GraphLoader.edgeListFile(sc, fname,
>> minEdgePartitions =
>> numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER)
>>
>> The error I am getting while executing this is:
>> Exception in thread "main" java.lang.UnsupportedOperationException:
>> Cannot change storage level of an RDD after it was already assigned a level
>>
>>
>> Then I looked into the GraphLoader class. I know that in the latest
>> version of spark support for setting persistence level is provided in this
>> class. Please suggest a workaround for spark 1.0.0 as I do not have the
>> option to shift to latest release.
>>
>> Note: I tried copying the GraphLoader class to my package as GraphLoader1
>> importing
>>
>> package com.cloudera.xyz
>>
>> import org.apache.spark.storage.StorageLevel
>> import org.apache.spark.graphx._
>> import org.apache.spark.{Logging, SparkContext}
>> import org.apache.spark.graphx.impl._
>>
>> and then changing the persistence level to my suitability as
>> .persist(gStorageLevel) instead of .cache()
>>
>> But while compiling I am getting the following errors
>>
>> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl
>> cannot be accessed in package org.apache.spark.graphx.impl
>> [INFO] val builder = new EdgePartitionBuilder[Int, Int]
>>
>> I am also attaching the file with the mail. Maybe this way of doing thing
>> is not possible.
>>
>>
>> Please suggest some workarounds so that I can set persistence level of my
>> graph to MEMORY_AND_DISK_SER for the graph I read from edge file list
>>
>
>
>
> --
> Arpit Kumar
> Fourth Year Undergraduate
> Department of Computer Science and Engineering
> Indian Institute of Technology, Kharagpur
>
>
>
--
Arpit Kumar
Fourth Year Undergraduate
Department of Computer Science and Engineering
Indian Institute of Technology, Kharagpur
Re: How to set persistence level of graph in GraphX in spark 1.0.0
Posted by Yifan LI <ia...@gmail.com>.
Hi Arpit,
To try this:
val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK,
vertexStorageLevel = StorageLevel.MEMORY_AND_DISK)
Best,
Yifan LI
On 28 Oct 2014, at 11:17, Arpit Kumar <ar...@gmail.com> wrote:
> Any help regarding this issue please?
>
> Regards,
> Arpit
>
> On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <ar...@gmail.com> wrote:
> Hi all,
> I am using the GrpahLoader class to load graphs from edge list files. But then I need to change the storage level of the graph to some other thing than MEMORY_ONLY.
>
> val graph = GraphLoader.edgeListFile(sc, fname,
> minEdgePartitions = numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER)
>
> The error I am getting while executing this is:
> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot change storage level of an RDD after it was already assigned a level
>
>
> Then I looked into the GraphLoader class. I know that in the latest version of spark support for setting persistence level is provided in this class. Please suggest a workaround for spark 1.0.0 as I do not have the option to shift to latest release.
>
> Note: I tried copying the GraphLoader class to my package as GraphLoader1 importing
>
> package com.cloudera.xyz
>
> import org.apache.spark.storage.StorageLevel
> import org.apache.spark.graphx._
> import org.apache.spark.{Logging, SparkContext}
> import org.apache.spark.graphx.impl._
>
> and then changing the persistence level to my suitability as
> .persist(gStorageLevel) instead of .cache()
>
> But while compiling I am getting the following errors
>
> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl cannot be accessed in package org.apache.spark.graphx.impl
> [INFO] val builder = new EdgePartitionBuilder[Int, Int]
>
> I am also attaching the file with the mail. Maybe this way of doing thing is not possible.
>
>
> Please suggest some workarounds so that I can set persistence level of my graph to MEMORY_AND_DISK_SER for the graph I read from edge file list
>
>
>
> --
> Arpit Kumar
> Fourth Year Undergraduate
> Department of Computer Science and Engineering
> Indian Institute of Technology, Kharagpur
Re: How to set persistence level of graph in GraphX in spark 1.0.0
Posted by Arpit Kumar <ar...@gmail.com>.
Any help regarding this issue please?
Regards,
Arpit
On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <ar...@gmail.com> wrote:
> Hi all,
> I am using the GrpahLoader class to load graphs from edge list files. But
> then I need to change the storage level of the graph to some other thing
> than MEMORY_ONLY.
>
> val graph = GraphLoader.edgeListFile(sc, fname,
> minEdgePartitions =
> numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER)
>
> The error I am getting while executing this is:
> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot
> change storage level of an RDD after it was already assigned a level
>
>
> Then I looked into the GraphLoader class. I know that in the latest
> version of spark support for setting persistence level is provided in this
> class. Please suggest a workaround for spark 1.0.0 as I do not have the
> option to shift to latest release.
>
> Note: I tried copying the GraphLoader class to my package as GraphLoader1
> importing
>
> package com.cloudera.xyz
>
> import org.apache.spark.storage.StorageLevel
> import org.apache.spark.graphx._
> import org.apache.spark.{Logging, SparkContext}
> import org.apache.spark.graphx.impl._
>
> and then changing the persistence level to my suitability as
> .persist(gStorageLevel) instead of .cache()
>
> But while compiling I am getting the following errors
>
> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl
> cannot be accessed in package org.apache.spark.graphx.impl
> [INFO] val builder = new EdgePartitionBuilder[Int, Int]
>
> I am also attaching the file with the mail. Maybe this way of doing thing
> is not possible.
>
>
> Please suggest some workarounds so that I can set persistence level of my
> graph to MEMORY_AND_DISK_SER for the graph I read from edge file list
>
--
Arpit Kumar
Fourth Year Undergraduate
Department of Computer Science and Engineering
Indian Institute of Technology, Kharagpur