You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jk...@apache.org on 2016/01/26 07:41:56 UTC
spark git commit: [SPARK-12834] Change ser/de of JavaArray and
JavaList
Repository: spark
Updated Branches:
refs/heads/master b66afdeb5 -> ae47ba718
[SPARK-12834] Change ser/de of JavaArray and JavaList
https://issues.apache.org/jira/browse/SPARK-12834
We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780
Author: Xusen Yin <yi...@gmail.com>
Closes #10772 from yinxusen/SPARK-12834.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ae47ba71
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ae47ba71
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ae47ba71
Branch: refs/heads/master
Commit: ae47ba718a280fc12720a71b981c38dbe647f35b
Parents: b66afde
Author: Xusen Yin <yi...@gmail.com>
Authored: Mon Jan 25 22:41:52 2016 -0800
Committer: Joseph K. Bradley <jo...@databricks.com>
Committed: Mon Jan 25 22:41:52 2016 -0800
----------------------------------------------------------------------
.../org/apache/spark/mllib/api/python/PythonMLLibAPI.scala | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ae47ba71/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 05f9a76..088ec6a 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -1490,7 +1490,11 @@ private[spark] object SerDe extends Serializable {
initialize()
def dumps(obj: AnyRef): Array[Byte] = {
- new Pickler().dumps(obj)
+ obj match {
+ // Pickler in Python side cannot deserialize Scala Array normally. See SPARK-12834.
+ case array: Array[_] => new Pickler().dumps(array.toSeq.asJava)
+ case _ => new Pickler().dumps(obj)
+ }
}
def loads(bytes: Array[Byte]): AnyRef = {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org