You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/07/07 01:02:13 UTC

[GitHub] spark pull request #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector shoul...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/18557

    [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should support `appendFloats` for array

    ## What changes were proposed in this pull request?
    
    This PR aims to add a missing `appendFloats` API for array into **ColumnVector** class. For double type, there is `appendDoubles` for array [here](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java#L818-L824).
    
    ## How was this patch tested?
    
    Pass the Jenkins with a newly added test case.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-20566-BRANCH-2.2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18557.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18557
    
----
commit 39839bf5b70aab603e538d424cda00ec7cde1402
Author: Dongjoon Hyun <do...@apache.org>
Date:   2017-05-04T13:04:15Z

    [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should support `appendFloats` for array
    
    This PR aims to add a missing `appendFloats` API for array into **ColumnVector** class. For double type, there is `appendDoubles` for array [here](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java#L818-L824).
    
    Pass the Jenkins with a newly added test case.
    
    Author: Dongjoon Hyun <do...@apache.org>
    
    Closes #17836 from dongjoon-hyun/SPARK-20566.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    Yep. It's totally internal officially.
    
    What I meant with `performance issue` is 3rd party can still use it and there might be a performance gap between `float` and `double`.
    
    I'll close this PR. Thank you again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    @dongjoon-hyun Is there any reason to backport this to previous versions? This is because we had such [a discussion](https://github.com/apache/spark/pull/17836#pullrequestreview-35957231).
    Obviously, it makes sense to support the latest one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    **[Test build #79306 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79306/testReport)** for PR 18557 at commit [`39839bf`](https://github.com/apache/spark/commit/39839bf5b70aab603e538d424cda00ec7cde1402).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    Hi, @kiszk .
    I think this is a bug fix of `ColumnVector` as described in [SPARK-20566](https://issues.apache.org/jira/browse/SPARK-20566).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    I know that 'there is no usage of this API internally in Spark 2.2', but it's only for 2.2.0.
    My reason was any 3rd party Spark library cannot use `ColumnVector` for `float` type in Spark 2.2.1+.
    
    Anyway, @cloud-fan changes the bug type. If that means backporting is not allowed for this patch, I have no objection for the community decision.
    
    So, @kiszk and @cloud-fan . Given that, may I close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by cloud-fan <gi...@git.apache.org>.

Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    `ColumnVector` is total internal in Spark 2.2, so there won't be 3rd party Spark library issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79306/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    BTW, thank you for swift reviews and feedbacks on my PR. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    We have not seen any failure in test suites. And, [there is no usage of this API](https://github.com/apache/spark/pull/17836#discussion_r114488839) in Spark 2.2.
    
    Does this missing cause any failure of test or application program? If so, it is good to put a sample program in this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    Hi, @cloud-fan .
    This is the backport for #17836 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18557: [SPARK-20566][SQL][BRANCH-2.2] ColumnVector should suppo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18557
  
    **[Test build #79306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79306/testReport)** for PR 18557 at commit [`39839bf`](https://github.com/apache/spark/commit/39839bf5b70aab603e538d424cda00ec7cde1402).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org