You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2021/07/06 12:25:58 UTC
[spark] branch master updated: [SPARK-36009][GRAPHX] Add missing
GraphX classes to registerKryoClasses util method
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2fff060 [SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method
2fff060 is described below
commit 2fff060b6d04e984352543efe9b8077b3c5d1c06
Author: Matthew Jones <ma...@vitaler.com>
AuthorDate: Tue Jul 6 07:25:22 2021 -0500
[SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method
### What changes were proposed in this pull request?
Add `ShippableVertexPartition` and `RoutingTablePartition` to the classes registered with Kryo in `GraphXUtils.registerKryoClasses`.
### Why are the changes needed?
`VertexRDDImpl` uses an `RDD[ShippableVertexPartition[VD]]` however, `GraphXUtils.registerKryoClasses` does not register `ShippableVertexPartition`. This means when running with `spark.kryo.registrationRequired` set to `true`, we get a "Class is not registered" exception. This is an issue as it prevents other unregistered classes from being discovered using `spark.kryo.registrationRequired` as the first unregistered class found halts the whole job. It also potentially decreases the se [...]
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested manually by running with `spark.kryo.registrationRequired` set to `true` and verifying no "Class is not registered" exception was thrown.
Closes #32973 from matthewrj/bug/register-graphx-classes.
Lead-authored-by: Matthew Jones <ma...@vitaler.com>
Co-authored-by: Matthew Jones <ml...@gmail.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala b/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala
index 4ff5b02..39822c4 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala
@@ -35,6 +35,8 @@ object GraphXUtils {
classOf[Edge[Object]],
classOf[(VertexId, Object)],
classOf[EdgePartition[Object, Object]],
+ classOf[ShippableVertexPartition[Object]],
+ classOf[RoutingTablePartition],
classOf[BitSet],
classOf[VertexIdToIndexMap],
classOf[VertexAttributeBlock[Object]],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org