You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2016/06/11 00:17:50 UTC
spark git commit: [SPARK-15884][SPARKR][SQL] Overriding stringArgs in
MapPartitionsInR
Repository: spark
Updated Branches:
refs/heads/master 2022afe57 -> 54f758b5f
[SPARK-15884][SPARKR][SQL] Overriding stringArgs in MapPartitionsInR
## What changes were proposed in this pull request?
As discussed in https://github.com/apache/spark/pull/12836
we need to override stringArgs method in MapPartitionsInR in order to avoid too large strings generated by "stringArgs" method based on the input arguments.
In this case exclude some of the input arguments: serialized R objects.
## How was this patch tested?
Existing test cases
Author: Narine Kokhlikyan <na...@gmail.com>
Closes #13610 from NarineK/dapply_MapPartitionsInR_stringArgs.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/54f758b5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/54f758b5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/54f758b5
Branch: refs/heads/master
Commit: 54f758b5fc60ecb0da6b191939a72ef5829be38c
Parents: 2022afe
Author: Narine Kokhlikyan <na...@gmail.com>
Authored: Fri Jun 10 17:17:47 2016 -0700
Committer: Cheng Lian <li...@databricks.com>
Committed: Fri Jun 10 17:17:47 2016 -0700
----------------------------------------------------------------------
.../org/apache/spark/sql/catalyst/plans/logical/object.scala | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/54f758b5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
index 55d8adf..78e8822 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
@@ -141,6 +141,9 @@ case class MapPartitionsInR(
outputObjAttr: Attribute,
child: LogicalPlan) extends ObjectConsumer with ObjectProducer {
override lazy val schema = outputSchema
+
+ override protected def stringArgs: Iterator[Any] = Iterator(inputSchema, outputSchema,
+ outputObjAttr, child)
}
object MapElements {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org