You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2016/05/21 21:41:05 UTC

[GitHub] spark pull request: [SPARK-15466][SQL] Make `SparkSession` as the ...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/13245

    [SPARK-15466][SQL] Make `SparkSession` as the entry point to programming with RDD too

    ## What changes were proposed in this pull request?
    
    `SparkSession` greatly reduces the number of concepts which Spark users must know. Currently, `SparkSession` is defined as the entry point to programming Spark with the Dataset and DataFrame API. And, we can easily get `RDD` by calling `Dataset.rdd` or `DataFrame.rdd`, too.
    
    However, many usages (including examples) are observed to extract `SparkSession.sparkContext` and keep it as own variable to call `parallelize`.
    
    If `SparkSession` supports RDD seamlessly too, it would be great for usability. We can do this by simply adding `parallelize` API.
    
    **Example**
    ```scala
     object SparkPi {
       def main(args: Array[String]) {
         val spark = SparkSession
           .builder
           .appName("Spark Pi")
           .getOrCreate()
    -    val sc = spark.sparkContext
         val slices = if (args.length > 0) args(0).toInt else 2
         val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    -    val count = sc.parallelize(1 until n, slices).map { i =>
    +    val count = spark.parallelize(1 until n, slices).map { i =>
         val count = spark.parallelize(1 until n, slices).map { i =>
           val x = random * 2 - 1
           val y = random * 2 - 1
           if (x*x + y*y < 1) 1 else 0
         }.reduce(_ + _)
         println("Pi is roughly " + 4.0 * count / n)
         spark.stop()
       }
     }
    ```
    
    ```python
     spark = SparkSession\
       .builder\
       .appName("PythonPi")\
       .getOrCreate()
    
    - sc = spark._sc
    -
     partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
     n = 100000 * partitions
    
     def f(_):
       x = random() * 2 - 1
       y = random() * 2 - 1
       return 1 if x ** 2 + y ** 2 < 1 else 0
    
    -count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
     count = spark.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
     print("Pi is roughly %f" % (4.0 * count / n))
    
     spark.stop()
    ```
    
    ## How was this patch tested?
    
    Pass the Jenkins test (with new python test) and also manual.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-15466

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13245.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13245
    
----
commit 810f08a666c5d14a2178e329b7c1727603be485e
Author: Dongjoon Hyun <do...@apache.org>
Date:   2016-05-21T21:36:04Z

    [SPARK-15466][SQL] Make `SparkSession` as the entry point to programming with RDD too

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15466][SQL] Make `SparkSession` as the ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13245#issuecomment-220809398
  
    **[Test build #59085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59085/consoleFull)** for PR 13245 at commit [`4f6a69e`](https://github.com/apache/spark/commit/4f6a69e75d3c96f3b2ed9d93edf8d1bf958acf1c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15466][SQL] Make `SparkSession` as the ...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun closed the pull request at:

    https://github.com/apache/spark/pull/13245


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15466][SQL] Make `SparkSession` as the ...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the pull request:

    https://github.com/apache/spark/pull/13245#issuecomment-220813421
  
    I see. Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15466][SQL] Make `SparkSession` as the ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13245#issuecomment-220802722
  
    **[Test build #59082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59082/consoleFull)** for PR 13245 at commit [`65f9746`](https://github.com/apache/spark/commit/65f9746362ac6fb227a5c8ff59717852b5ae87c4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org