You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/10 20:56:28 UTC

[GitHub] [spark] hvanhovell opened a new pull request #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

hvanhovell opened a new pull request #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530
 
 
   ### What changes were proposed in this pull request?
   This is a small follow-up for https://github.com/apache/spark/pull/27400. This PR makes an empty `LocalTableScanExec` return an `RDD` without partitions.
   
   ### Why are the changes needed?
   It is a bit unexpected that the RDD contains partitions if there is not work to do. It also can save a bit of work when this is used in a more complex plan.
   
   ### Does this PR introduce any user-facing change?
   No
   
   ### How was this patch tested?
   Added test to `SparkPlanSuite`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351718
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22941/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584816597
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23019/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584816583
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584816583
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351704
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584438763
 
 
   **[Test build #118179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118179/testReport)** for PR 27530 at commit [`5d5fd4f`](https://github.com/apache/spark/commit/5d5fd4f19963af01b8b2c4e5117a869916fcc122).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584438958
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584931866
 
 
   **[Test build #118260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118260/testReport)** for PR 27530 at commit [`6d46dec`](https://github.com/apache/spark/commit/6d46dec2061054da6b8c041b5649d0778d2e9471).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351097
 
 
   **[Test build #118179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118179/testReport)** for PR 27530 at commit [`5d5fd4f`](https://github.com/apache/spark/commit/5d5fd4f19963af01b8b2c4e5117a869916fcc122).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584816597
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23019/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584438964
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118179/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584932962
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118260/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584815900
 
 
   **[Test build #118260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118260/testReport)** for PR 27530 at commit [`6d46dec`](https://github.com/apache/spark/commit/6d46dec2061054da6b8c041b5649d0778d2e9471).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584438964
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118179/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584932952
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584932952
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584976583
 
 
   Merged to master, and branch-3.0 to consistent with #27400.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#discussion_r377544690
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala
 ##########
 @@ -45,10 +45,14 @@ case class LocalTableScanExec(
     }
   }
 
-  private lazy val numParallelism: Int = math.min(math.max(unsafeRows.length, 1),
-    sqlContext.sparkContext.defaultParallelism)
-
-  private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows, numParallelism)
+  @transient private lazy val rdd: RDD[InternalRow] = {
+    if (rows.isEmpty) {
 
 Review comment:
   Maybe, `unsafeRows.isEmpty`? Otherwise I have to look at the difference between `unsafeRows` and `rows`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#discussion_r377545495
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala
 ##########
 @@ -45,10 +45,14 @@ case class LocalTableScanExec(
     }
   }
 
-  private lazy val numParallelism: Int = math.min(math.max(unsafeRows.length, 1),
-    sqlContext.sparkContext.defaultParallelism)
-
-  private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows, numParallelism)
+  @transient private lazy val rdd: RDD[InternalRow] = {
+    if (rows.isEmpty) {
+      sqlContext.sparkContext.emptyRDD
+    } else {
+      val numSlices = math.min(unsafeRows.length, sqlContext.sparkContext.defaultParallelism)
+      sqlContext.sparkContext.parallelize(unsafeRows, numSlices)
 
 Review comment:
   Just in case, does it make sense to put this code (handling empty rows) inside of `parallelize`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#discussion_r377569895
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala
 ##########
 @@ -45,10 +45,14 @@ case class LocalTableScanExec(
     }
   }
 
-  private lazy val numParallelism: Int = math.min(math.max(unsafeRows.length, 1),
-    sqlContext.sparkContext.defaultParallelism)
-
-  private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows, numParallelism)
+  @transient private lazy val rdd: RDD[InternalRow] = {
+    if (rows.isEmpty) {
+      sqlContext.sparkContext.emptyRDD
+    } else {
+      val numSlices = math.min(unsafeRows.length, sqlContext.sparkContext.defaultParallelism)
+      sqlContext.sparkContext.parallelize(unsafeRows, numSlices)
 
 Review comment:
   `parallelize` need to respect the `numSlices` parameter, even if the data is empty.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] hvanhovell commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#discussion_r377853444
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala
 ##########
 @@ -45,10 +45,14 @@ case class LocalTableScanExec(
     }
   }
 
-  private lazy val numParallelism: Int = math.min(math.max(unsafeRows.length, 1),
-    sqlContext.sparkContext.defaultParallelism)
-
-  private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows, numParallelism)
+  @transient private lazy val rdd: RDD[InternalRow] = {
+    if (rows.isEmpty) {
 
 Review comment:
   This way we avoid materializing the `unsafeRows` lazy val.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351704
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351097
 
 
   **[Test build #118179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118179/testReport)** for PR 27530 at commit [`5d5fd4f`](https://github.com/apache/spark/commit/5d5fd4f19963af01b8b2c4e5117a869916fcc122).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584815900
 
 
   **[Test build #118260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118260/testReport)** for PR 27530 at commit [`6d46dec`](https://github.com/apache/spark/commit/6d46dec2061054da6b8c041b5649d0778d2e9471).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584438958
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27530: [SPARK-30780][SQL] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584932962
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118260/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27530: [SPARK-30780] Empty LocalTableScan should use RDD without partitions
URL: https://github.com/apache/spark/pull/27530#issuecomment-584351718
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22941/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org