You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mgaido91 <gi...@git.apache.org> on 2017/12/05 13:02:44 UTC

[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

GitHub user mgaido91 opened a pull request:

    https://github.com/apache/spark/pull/19896

    [SPARK-22693][SQL] InSet and CreateNamedStruct should not use global variables

    ## What changes were proposed in this pull request?
    
    InSet and CreateNamedStruct are using a global variable which is not needed. This can generate some unneeded entries in the constant pool.
    
    The PR removes the unnecessary mutable states and makes them local variables.
    
    ## How was this patch tested?
    
    added UT


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mgaido91/spark SPARK-22693

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19896.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19896
    
----
commit ac7131fbce4c9c83e159481c38c0833e211c0c58
Author: Marco Gaido <mg...@hortonworks.com>
Date:   2017-12-05T12:31:12Z

    [SPARK-22693][SQL] InSet and CreateNamedStruct should not use global variables

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155218004
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    oh it didn't, but it can, you know my point...


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155218181
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    why? It is creating it every time (`new Object[${valExprs.size}]`), the only thing which is reused is the pointer to the array.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84564/testReport)** for PR 19896 at commit [`36537d8`](https://github.com/apache/spark/commit/36537d880700f6725fcb9221519a74a53f6b7748).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84564/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    thank you all, please wait for #19910 because if it is not feasible to reuse the array object as @viirya 
     says, I will add back here the change to avoid that global variable. Thanks.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84484/testReport)** for PR 19896 at commit [`ac7131f`](https://github.com/apache/spark/commit/ac7131fbce4c9c83e159481c38c0833e211c0c58).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84546/testReport)** for PR 19896 at commit [`91565d1`](https://github.com/apache/spark/commit/91565d14fe1834c6b24f617a7bba099fcd50cb2e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84484/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84552/testReport)** for PR 19896 at commit [`b5bd951`](https://github.com/apache/spark/commit/b5bd951db61994f0d93876cbc277af6519523194).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    @cloud-fan @kiszk @viirya may you please review this? Thanks


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84546/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84564/testReport)** for PR 19896 at commit [`36537d8`](https://github.com/apache/spark/commit/36537d880700f6725fcb9221519a74a53f6b7748).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84552/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155226100
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    yea sounds good


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    I added back the changes for CreateNamedStruct. May you review this part again too? Thanks.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet should no...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Thanks! Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155222527
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    I mean it can reuse the array...


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84484/testReport)** for PR 19896 at commit [`ac7131f`](https://github.com/apache/spark/commit/ac7131fbce4c9c83e159481c38c0833e211c0c58).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155217583
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
    --- End diff --
    
    nit: please keep the parameter name
    ```
    ctx.splitExpressionsWithCurrentInputs(
      expressions = valCodes,
      funcName = ...
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155224703
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    I see what you mean now. Since currently this is not done, what should we do? Should I create a new PR to reuse the array and remove this from here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] CreateNamedStruct and InSet sh...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19896


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet should not use global variables

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84552/testReport)** for PR 19896 at commit [`b5bd951`](https://github.com/apache/spark/commit/b5bd951db61994f0d93876cbc277af6519523194).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct sh...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19896#discussion_r155217888
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ---
    @@ -356,22 +356,25 @@ case class CreateNamedStruct(children: Seq[Expression]) extends CreateNamedStruc
       override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val rowClass = classOf[GenericInternalRow].getName
         val values = ctx.freshName("values")
    -    ctx.addMutableState("Object[]", values, s"$values = null;")
    +    val valCodes = valExprs.zipWithIndex.map { case (e, i) =>
    +      val eval = e.genCode(ctx)
    +      s"""
    +         |${eval.code}
    +         |if (${eval.isNull}) {
    +         |  $values[$i] = null;
    +         |} else {
    +         |  $values[$i] = ${eval.value};
    +         |}
    +       """.stripMargin
    +    }
         val valuesCode = ctx.splitExpressionsWithCurrentInputs(
    -      valExprs.zipWithIndex.map { case (e, i) =>
    -        val eval = e.genCode(ctx)
    -        s"""
    -          ${eval.code}
    -          if (${eval.isNull}) {
    -            $values[$i] = null;
    -          } else {
    -            $values[$i] = ${eval.value};
    -          }"""
    -      })
    +      valCodes,
    +      "createNamedStruct",
    +      "Object[]" -> values :: Nil)
     
         ev.copy(code =
           s"""
    -         |$values = new Object[${valExprs.size}];
    +         |Object[] $values = new Object[${valExprs.size}];
    --- End diff --
    
    actually I'm not very sure about this. The previous code can avoid creating this array every time.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19896: [SPARK-22693][SQL] InSet and CreateNamedStruct should no...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19896
  
    **[Test build #84546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84546/testReport)** for PR 19896 at commit [`91565d1`](https://github.com/apache/spark/commit/91565d14fe1834c6b24f617a7bba099fcd50cb2e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `    case class ReqAndHandler(req: Request, handler: MemberHandler)`
      * `trait TypeCoercionRule extends Rule[LogicalPlan] with Logging `
      * `case class AnalysisBarrier(child: LogicalPlan) extends LeafNode `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org