You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2015/12/04 21:35:04 UTC

[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

GitHub user gatorsmile opened a pull request:

    https://github.com/apache/spark/pull/10149

    [SPARK-12150] [SQL] [Minor] Add range API without specifying the slice number

    For usability, add another sqlContext.range() method. Users can specify start, end, and step without specifying the slice number. The slice number is based on the sparkContext's defaultParallelism. It just makes consistent with the RDD range API. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gatorsmile/spark range

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/10149.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10149
    
----
commit 8c4bd8351f79db2ce2aebc8a641442ba882295b8
Author: gatorsmile <ga...@gmail.com>
Date:   2015-12-04T20:23:36Z

    range API with a default partition number

commit 6655b9d9515819cf81844c63c7105eb59882be12
Author: gatorsmile <ga...@gmail.com>
Date:   2015-12-04T20:25:52Z

    2.0->1.6

commit 72860c4e93de38da18ee13e46368493d04819094
Author: gatorsmile <ga...@gmail.com>
Date:   2015-12-04T20:27:02Z

    Merge remote-tracking branch 'upstream/master' into range

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-163093870
  
    @marmbrus @cloud-fan This PR changes the external API. Not sure if this will be merged or we should revisit it after the release of 1.6? Thank you!  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-162098991
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47206/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-164942006
  
    I will make a try this week. Thanks for your insights! : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-164996025
  
    Thanks! Will do my best. : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:

    https://github.com/apache/spark/pull/10149


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-162077838
  
    **[Test build #47206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47206/consoleFull)** for PR 10149 at commit [`72860c4`](https://github.com/apache/spark/commit/72860c4e93de38da18ee13e46368493d04819094).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-164874665
  
    This seems reasonable, but if we are going to do work on `sqlContext.range` it would be nice if we could take care of a bigger issue.  Right now this code is much slower than it needs to be.
    
    ![screen shot 2015-12-08 at 4 20 59 pm](https://cloud.githubusercontent.com/assets/527/11821919/c6df5dec-a320-11e5-82b3-265ff0ecf73a.png)
    
    I think that is is because we are boxing and creating a new Row for each element.  Instead, we should consider creating an actual logical/physical operator for range.  the implementation of doExecute() can mirror the code in `sparkContext.range`, but would allocate one `UnsafeRow` per partition and just call `setLong` for each element (reusing the same buffer).
    
    Does that make sense?
    
    Benchmark code here: https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/DatasetPerformance.scala


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-164944914
  
    Cool, let me know if you run into any problems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-164877767
  
    That sounds an interesting work! Let me think about it. 
    
    Do you want me close this PR? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-162098834
  
    **[Test build #47206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47206/consoleFull)** for PR 10149 at commit [`72860c4`](https://github.com/apache/spark/commit/72860c4e93de38da18ee13e46368493d04819094).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `  public abstract static class PrefixComputer `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12150] [SQL] [Minor] Add range API with...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10149#issuecomment-162098990
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org