You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by da...@apache.org on 2015/09/17 18:21:27 UTC
spark git commit: [SPARK-10459] [SQL] Do not need to have
ConvertToSafe for PythonUDF
Repository: spark
Updated Branches:
refs/heads/master e51345e1e -> 2a508df20
[SPARK-10459] [SQL] Do not need to have ConvertToSafe for PythonUDF
JIRA: https://issues.apache.org/jira/browse/SPARK-10459
As mentioned in the JIRA, `PythonUDF` actually could process `UnsafeRow`.
Specially, the rows in `childResults` in `BatchPythonEvaluation` will be projected to a `MutableRow`. So I think we can enable `canProcessUnsafeRows` for `BatchPythonEvaluation` and get rid of redundant `ConvertToSafe`.
Author: Liang-Chi Hsieh <vi...@appier.com>
Closes #8616 from viirya/pyudf-unsafe.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a508df2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a508df2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2a508df2
Branch: refs/heads/master
Commit: 2a508df20d03b3d4a3c05b65fb02d849bc080ef9
Parents: e51345e
Author: Liang-Chi Hsieh <vi...@appier.com>
Authored: Thu Sep 17 09:21:21 2015 -0700
Committer: Davies Liu <da...@gmail.com>
Committed: Thu Sep 17 09:21:21 2015 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/sql/execution/pythonUDFs.scala | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/2a508df2/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
index 5a58d84..d0411da 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
@@ -337,6 +337,10 @@ case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child:
def children: Seq[SparkPlan] = child :: Nil
+ override def outputsUnsafeRows: Boolean = false
+ override def canProcessUnsafeRows: Boolean = true
+ override def canProcessSafeRows: Boolean = true
+
protected override def doExecute(): RDD[InternalRow] = {
val childResults = child.execute().map(_.copy())
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org