You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mgaido91 <gi...@git.apache.org> on 2017/12/05 13:02:44 UTC
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/19896
[SPARK-22693][SQL] InSet and CreateNamedStruct should not use global variables
## What changes were proposed in this pull request?
InSet and CreateNamedStruct are using a global variable which is not needed. This can generate some unneeded entries in the constant pool.
The PR removes the unnecessary mutable states and makes them local variables.
## How was this patch tested?
added UT
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-22693
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19896.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19896
----
commit ac7131fbce4c9c83e159481c38c0833e211c0c58
Author: Marco Gaido <mg...@hortonworks.com>
Date: 2017-12-05T12:31:12Z
[SPARK-22693][SQL] InSet and CreateNamedStruct should not use global variables
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155218004
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
oh it didn't, but it can, you know my point...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19896
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155218181
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
why? It is creating it every time (`new Object[${valExprs.size}]`), the only thing which is reused is the pointer to the array.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84564/testReport)** for PR 19896 at commit [`36537d8`](https://github.com/apache/spark/commit/36537d880700f6725fcb9221519a74a53f6b7748).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84564/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/19896
thank you all, please wait for #19910 because if it is not feasible to reuse the array object as @viirya
says, I will add back here the change to avoid that global variable. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84484/testReport)** for PR 19896 at commit [`ac7131f`](https://github.com/apache/spark/commit/ac7131fbce4c9c83e159481c38c0833e211c0c58).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19896
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84546/testReport)** for PR 19896 at commit [`91565d1`](https://github.com/apache/spark/commit/91565d14fe1834c6b24f617a7bba099fcd50cb2e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84484/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84552/testReport)** for PR 19896 at commit [`b5bd951`](https://github.com/apache/spark/commit/b5bd951db61994f0d93876cbc277af6519523194).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19896
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/19896
@cloud-fan @kiszk @viirya may you please review this? Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84546/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84564/testReport)** for PR 19896 at commit [`36537d8`](https://github.com/apache/spark/commit/36537d880700f6725fcb9221519a74a53f6b7748).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84552/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19896
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155226100
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
yea sounds good
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/19896
I added back the changes for CreateNamedStruct. May you review this part again too? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19896
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155222527
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
I mean it can reuse the array...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84484/testReport)** for PR 19896 at commit [`ac7131f`](https://github.com/apache/spark/commit/ac7131fbce4c9c83e159481c38c0833e211c0c58).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19896
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155217583
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
--- End diff --
nit: please keep the parameter name
```
ctx.splitExpressionsWithCurrentInputs(
expressions = valCodes,
funcName = ...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155224703
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
I see what you mean now. Since currently this is not done, what should we do? Should I create a new PR to reuse the array and remove this from here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet sh...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19896
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84552/testReport)** for PR 19896 at commit [`b5bd951`](https://github.com/apache/spark/commit/b5bd951db61994f0d93876cbc277af6519523194).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19896#discussion_r155217888
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
@@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val rowClass = classOf[GenericInternalRow].getName
val values = ctx.freshName("values")
- ctx.addMutableState("Object[]", values, s"$values = null;")
+ val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
+ val eval = e.genCode(ctx)
+ s"""
+ |${eval.code}
+ |if (${eval.isNull}) {
+ | $values[$i] = null;
+ |} else {
+ | $values[$i] = ${eval.value};
+ |}
+ """.stripMargin
+ }
val valuesCode = ctx.splitExpressionsWithCurrentInputs(
- valExprs.zipWithIndex.map { case (e, i) =>
- val eval = e.genCode(ctx)
- s"""
- ${eval.code}
- if (${eval.isNull}) {
- $values[$i] = null;
- } else {
- $values[$i] = ${eval.value};
- }"""
- })
+ valCodes,
+ "createNamedStruct",
+ "Object[]" -> values :: Nil)
ev.copy(code =
s"""
- |$values = new Object[${valExprs.size}];
+ |Object[] $values = new Object[${valExprs.size}];
--- End diff --
actually I'm not very sure about this. The previous code can avoid creating this array every time.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19896
**[Test build #84546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84546/testReport)** for PR 19896 at commit [`91565d1`](https://github.com/apache/spark/commit/91565d14fe1834c6b24f617a7bba099fcd50cb2e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class ReqAndHandler(req: Request, handler: MemberHandler)`
* `trait TypeCoercionRule extends Rule[LogicalPlan] with Logging `
* `case class AnalysisBarrier(child: LogicalPlan) extends LeafNode `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org