You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jliwork <gi...@git.apache.org> on 2015/10/23 11:16:13 UTC

[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

GitHub user jliwork opened a pull request:

    https://github.com/apache/spark/pull/9247

    [SPARK-11277] [SQL] sort_array throws exception scala.MatchError

    I'm new to spark. I was trying out the sort_array function then hit this exception. I looked into the spark source code. I found the root cause is that sort_array does not check for an array of NULLs. It's not meaningful to sort an array of entirely NULLs anyway.
    
    I'm adding a check on the input array type to SortArray. If the array consists of NULLs entirely, there is no need to sort such array. I have also added a test case for this.
    
    Please help to review my fix. Thanks!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jliwork/spark SPARK-11277

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/9247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #9247
    
----
commit 998b8b84f2667b1d16c732d63ba70ab6bfc5b56e
Author: Jia Li <ji...@us.ibm.com>
Date:   2015-10-23T09:06:26Z

    [SPARK-11277] [SQL] sort_array throws exception scala.MatchError

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151066487
  
    We support sorting on structs due to an `ORDER BY`, I don't see why we shouldn't support it in arrays.  Its useful for computing things like argmax.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151035107
  
    I think we should follow hive, thus we also need to update the `checkInputDataTypes` to deny struct type.
    
    cc @yhuai @marmbrus , what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150526270
  
    ok to test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151340580
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150918873
  
    Hi @jliwork , thanks for working on it!
    But sorting on array of null type doesn't make sense to me, can you check the behaviour of other SQL system like Hive? And how about struct type? It's also order-able.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151356391
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150525067
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151030490
  
    @cloud-fan Thanks for the clarification. You have a very good point. This bug's exception occourred because SortArray's Comparators expect AtomicType only. According to source code docs, AtomicType 'represent everything that is not null, UDTs, arrays, structs, and maps'. My current fix takes care of NullType only. StructType still causes execption, so you're right :-)
    
    SortArray actually comes with checkInputDataTypes() which lets NullType and StructType slip through because it uses RowOrdering.isOrderable which returns true for AtomicType, NullType and StructType. I checked Hive's behavior for struct type element. For example, 
    
    select sort_array(array(struct(3, 2, 1), struct(2, 3, 4))) from src
    
    Hive does not support sort array with struct type involved. It returns error for this query, saying function SORT_ARRAY must be array<PRIMITIVE>, but array<struct<col1:int,col2:int,col3:int>> was found. 
    
    QUESTION - What do you expect the behavior of the query above for Spark SQL? I would prefer to match with Hive's behavior which is to block this kind of sorting on struct type elements since its use case does not seem very meaningful to me.
    
    Thanks,


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151356393
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44388/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151370701
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150526992
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9247#discussion_r43074419
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala ---
    @@ -64,6 +65,21 @@ class CollectionFunctionsSuite extends SparkFunSuite with ExpressionEvalHelper {
         checkEvaluation(new SortArray(a3, Literal(false)), Seq("b", "a", null))
     
         checkEvaluation(Literal.create(null, ArrayType(StringType)), null)
    +    checkEvaluation(new SortArray(a4), Seq(null, null))
    +
    +    val typeAS = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
    +    val arrayStruct = Literal.create(Seq(create_row(2), create_row(1)), typeAS)
    +    val sorted = new SortArray(arrayStruct)
    +
    +    def getArrayStructFields(expr: Expression, fieldName: String): GetArrayStructFields = {
    +      expr.dataType match {
    +        case ArrayType(StructType(fields), containsNull) =>
    +          val field = fields.find(_.name == fieldName).get
    +          GetArrayStructFields(expr, field, fields.indexOf(field), fields.length, containsNull)
    +      }
    +    }
    +
    +    checkEvaluation(getArrayStructFields(sorted, "a"), Seq(1, 2))
    --- End diff --
    
    I was just trying to find a way to compare the results. Your approach is much simpler. I have updated the test case based on your comment. Thank you for your comment! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151297377
  
    @marmbrus @cloud-fan I now got better understanding on the struct type after looking into the code. Let me give a try to enhance sort_array to support struct type. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151356075
  
    **[Test build #44388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44388/consoleFull)** for PR 9247 at commit [`5ae83c4`](https://github.com/apache/spark/commit/5ae83c45eecc42df6eae59e8d62114ce0565414f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151329960
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/9247


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150546790
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44218/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150546789
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150526958
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151313869
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151329981
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151010243
  
    Ah, I don't mean something like sort_struct, but sort_array with struct type elements. Does hive support it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151313883
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151274333
  
    @marmbrus @cloud-fan Thank you for your comment. My current fix takes care of array of NullType. I should be able to give error to block sort_array on struct type as well if you like. I just looked into Spark's ORDER BY on struct type. Since I'm new to spark, however, I feel a little bit challenging for me to enhance sort_array to include struct type at this point. Would you be OK if I open another JIRA to support sort_array on struct type, so that someone in the community might be able to help?
    
    Thanks,


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150527270
  
    **[Test build #44218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44218/consoleFull)** for PR 9247 at commit [`998b8b8`](https://github.com/apache/spark/commit/998b8b84f2667b1d16c732d63ba70ab6bfc5b56e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151314576
  
    **[Test build #44385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44385/consoleFull)** for PR 9247 at commit [`a2bac0a`](https://github.com/apache/spark/commit/a2bac0a4e86f24240f237505b556190717403465).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151340487
  
    **[Test build #44385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44385/consoleFull)** for PR 9247 at commit [`a2bac0a`](https://github.com/apache/spark/commit/a2bac0a4e86f24240f237505b556190717403465).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151439866
  
    Thanks, merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151330425
  
    **[Test build #44388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44388/consoleFull)** for PR 9247 at commit [`5ae83c4`](https://github.com/apache/spark/commit/5ae83c45eecc42df6eae59e8d62114ce0565414f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9247#discussion_r43073001
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala ---
    @@ -64,6 +65,21 @@ class CollectionFunctionsSuite extends SparkFunSuite with ExpressionEvalHelper {
         checkEvaluation(new SortArray(a3, Literal(false)), Seq("b", "a", null))
     
         checkEvaluation(Literal.create(null, ArrayType(StringType)), null)
    +    checkEvaluation(new SortArray(a4), Seq(null, null))
    +
    +    val typeAS = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
    +    val arrayStruct = Literal.create(Seq(create_row(2), create_row(1)), typeAS)
    +    val sorted = new SortArray(arrayStruct)
    +
    +    def getArrayStructFields(expr: Expression, fieldName: String): GetArrayStructFields = {
    +      expr.dataType match {
    +        case ArrayType(StructType(fields), containsNull) =>
    +          val field = fields.find(_.name == fieldName).get
    +          GetArrayStructFields(expr, field, fields.indexOf(field), fields.length, containsNull)
    +      }
    +    }
    +
    +    checkEvaluation(getArrayStructFields(sorted, "a"), Seq(1, 2))
    --- End diff --
    
    Why get struct fields here?
    Can we test `checkEvaluation(new SortArray(arrayStruct), Seq(create_row(1), create_row(2)))`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151340581
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44385/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150546662
  
    **[Test build #44218 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44218/consoleFull)** for PR 9247 at commit [`998b8b8`](https://github.com/apache/spark/commit/998b8b84f2667b1d16c732d63ba70ab6bfc5b56e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-151341426
  
    Thank you all. I have updated my fix based on your comments. It should now be able to:
    1) skip sorting an array of NULLs 
    2) sort an array of StructType
    
    @cloud-fan @marmbrus @yhuai 
    Please let me know if you have any further questions or comments. Thanks again,



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...

Posted by jliwork <gi...@git.apache.org>.
Github user jliwork commented on the pull request:

    https://github.com/apache/spark/pull/9247#issuecomment-150992372
  
    @cloud-fan Thank you very much for your comment! Sorting on array of NULL type does not make sense to me, either. That's why my fix is to block this kind of sorting for sort_array. 
    
    1)I have checked Hive's behavior. Hive does not throws any exception when sorting an array of NULLs. It will simple return the array of NULLs back. With my fix spark's sort_array will return the exact same result as Hive. 
    2) Regarding sorting on struct, I could not find any existing UDF like sort_struct. Hive does not seems to offer such UDF, either. I'm not very familiar with dataframe's struct type as I'm new to Spark and still learning.
    
    Please let me know if you have any other questions or concerns. Thank again. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org