You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jk...@apache.org on 2016/01/27 09:33:07 UTC
spark git commit: [SPARK-12834][ML][PYTHON][BACKPORT] Change ser/de
of JavaArray and JavaList
Repository: spark
Updated Branches:
refs/heads/branch-1.6 85518eda4 -> 17d1071ce
[SPARK-12834][ML][PYTHON][BACKPORT] Change ser/de of JavaArray and JavaList
Backport of SPARK-12834 for branch-1.6
Original PR: https://github.com/apache/spark/pull/10772
Original commit message:
We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780
Author: Xusen Yin <yi...@gmail.com>
Closes #10941 from jkbradley/yinxusen-SPARK-12834-1.6.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/17d1071c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/17d1071c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/17d1071c
Branch: refs/heads/branch-1.6
Commit: 17d1071ce8945d056da145f64797d1d10529afc1
Parents: 85518ed
Author: Xusen Yin <yi...@gmail.com>
Authored: Wed Jan 27 00:32:52 2016 -0800
Committer: Joseph K. Bradley <jo...@databricks.com>
Committed: Wed Jan 27 00:32:52 2016 -0800
----------------------------------------------------------------------
.../org/apache/spark/mllib/api/python/PythonMLLibAPI.scala | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/17d1071c/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 29160a1..1714983 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -1464,7 +1464,11 @@ private[spark] object SerDe extends Serializable {
initialize()
def dumps(obj: AnyRef): Array[Byte] = {
- new Pickler().dumps(obj)
+ obj match {
+ // Pickler in Python side cannot deserialize Scala Array normally. See SPARK-12834.
+ case array: Array[_] => new Pickler().dumps(array.toSeq.asJava)
+ case _ => new Pickler().dumps(obj)
+ }
}
def loads(bytes: Array[Byte]): AnyRef = {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org