You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/03 16:31:19 UTC
[GitHub] [spark] wangyum opened a new pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
wangyum opened a new pull request #29641:
URL: https://github.com/apache/spark/pull/29641
### What changes were proposed in this pull request?
This pr make non-partitioned table metric should not have dynamic partition pruning time.
### Why are the changes needed?
It is useless for non-partitioned table.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test
Before this pr:
After this pr:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686774565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29641:
URL: https://github.com/apache/spark/pull/29641#discussion_r483649563
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
##########
@@ -384,6 +386,7 @@ abstract class DynamicPartitionPruningSuiteBase
assert(scan3.metrics("numFiles").value == partFilesNum)
assert(scan3.metrics("filesSize").value == partFilesSize)
assert(scan3.metrics("numPartitions").value === 1)
+ assert(scan3.metrics("pruningTime").value !== -1)
Review comment:
what is the difference between `!=` and `!==`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-687628357
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29641:
URL: https://github.com/apache/spark/pull/29641#discussion_r483649563
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
##########
@@ -384,6 +386,7 @@ abstract class DynamicPartitionPruningSuiteBase
assert(scan3.metrics("numFiles").value == partFilesNum)
assert(scan3.metrics("filesSize").value == partFilesSize)
assert(scan3.metrics("numPartitions").value === 1)
+ assert(scan3.metrics("pruningTime").value !== -1)
Review comment:
nit: what is the difference between `!=` and `!==`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686773739
**[Test build #128265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128265/testReport)** for PR 29641 at commit [`216f225`](https://github.com/apache/spark/commit/216f225bf8f2d92ba26cf02b83ed7f199f2a62a9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686859697
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29641:
URL: https://github.com/apache/spark/pull/29641#discussion_r483960837
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
##########
@@ -384,6 +386,7 @@ abstract class DynamicPartitionPruningSuiteBase
assert(scan3.metrics("numFiles").value == partFilesNum)
assert(scan3.metrics("filesSize").value == partFilesSize)
assert(scan3.metrics("numPartitions").value === 1)
+ assert(scan3.metrics("pruningTime").value !== -1)
Review comment:
Using `!==`can give more informative error message:
https://www.scalatest.org/getting_started_with_fun_suite
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686774565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686613506
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #29641:
URL: https://github.com/apache/spark/pull/29641
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686612816
**[Test build #128265 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128265/testReport)** for PR 29641 at commit [`216f225`](https://github.com/apache/spark/commit/216f225bf8f2d92ba26cf02b83ed7f199f2a62a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29641:
URL: https://github.com/apache/spark/pull/29641#discussion_r483960837
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
##########
@@ -384,6 +386,7 @@ abstract class DynamicPartitionPruningSuiteBase
assert(scan3.metrics("numFiles").value == partFilesNum)
assert(scan3.metrics("filesSize").value == partFilesSize)
assert(scan3.metrics("numPartitions").value === 1)
+ assert(scan3.metrics("pruningTime").value !== -1)
Review comment:
Using `!==`would give more informative error message:
https://www.scalatest.org/getting_started_with_fun_suite
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686612816
**[Test build #128265 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128265/testReport)** for PR 29641 at commit [`216f225`](https://github.com/apache/spark/commit/216f225bf8f2d92ba26cf02b83ed7f199f2a62a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29641: [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic partition pruning time
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29641:
URL: https://github.com/apache/spark/pull/29641#issuecomment-686613506
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org