You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/25 07:37:43 UTC
[GitHub] [spark] prakharjain09 opened a new pull request #28634: [SPARK-31810][TEST] Fix test where spark conf was modified using incorrect api
prakharjain09 opened a new pull request #28634:
URL: https://github.com/apache/spark/pull/28634
### What changes were proposed in this pull request?
Use the correct API in AlterTableRecoverPartition tests to modify the `RDD_PARALLEL_LISTING_THRESHOLD` conf.
### Why are the changes needed?
The existing AlterTableRecoverPartitions test modify the RDD_PARALLEL_LISTING_THRESHOLD as a SQLConf using the withSQLConf API. But since, this is not a SQLConf, it is not overridden and so the test doesn't end up testing the required behaviour.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
This is UT Fix. UTs are still passing after the fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] prakharjain09 commented on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
prakharjain09 commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633481802
cc - @srowen @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633811208
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28634: [SPARK-31810][TEST] Fix test where spark conf was modified using incorrect api
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633430139
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #28634:
URL: https://github.com/apache/spark/pull/28634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633688481
**[Test build #123085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123085/testReport)** for PR 28634 at commit [`f6900b3`](https://github.com/apache/spark/commit/f6900b32b361777ec32e27ec6a8df33c068b1f4d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633689329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] prakharjain09 edited a comment on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
prakharjain09 edited a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633481802
cc - @srowen @cloud-fan @Dooyoung-Hwang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28634:
URL: https://github.com/apache/spark/pull/28634#discussion_r429962358
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1203,14 +1203,24 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
test("alter table: recover partitions (sequential)") {
- withSQLConf(RDD_PARALLEL_LISTING_THRESHOLD.key -> "10") {
+ val oldRddParallelListingThreshold = spark.sparkContext.conf.get(
+ RDD_PARALLEL_LISTING_THRESHOLD)
+ try {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD.key, "10")
Review comment:
I have the impression that Spark conf is immutable. Are you sure this works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28634: [SPARK-31810][TEST] Fix test where spark conf was modified using incorrect api
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633429669
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28634: [SPARK-31810][TEST] Fix test where spark conf was modified using incorrect api
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633429669
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] prakharjain09 commented on a change in pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
prakharjain09 commented on a change in pull request #28634:
URL: https://github.com/apache/spark/pull/28634#discussion_r429966067
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1203,14 +1203,24 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
test("alter table: recover partitions (sequential)") {
- withSQLConf(RDD_PARALLEL_LISTING_THRESHOLD.key -> "10") {
+ val oldRddParallelListingThreshold = spark.sparkContext.conf.get(
+ RDD_PARALLEL_LISTING_THRESHOLD)
+ try {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD.key, "10")
Review comment:
Yes this works. I have verified this by printing the value of this conf inside the main code in AlterTableRecoverPartitions class. Both of the tests prints different values - 0 and 10.
Also there are many tests which modify conf using "sc.conf.set". Ex - TaskSetManagerSuite, RDDSuite etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] prakharjain09 edited a comment on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
prakharjain09 edited a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633481802
cc - @srowen @cloud-fan @Dooyoung-Hwang @holdenk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633581100
**[Test build #123085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123085/testReport)** for PR 28634 at commit [`f6900b3`](https://github.com/apache/spark/commit/f6900b32b361777ec32e27ec6a8df33c068b1f4d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633811208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28634: [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28634:
URL: https://github.com/apache/spark/pull/28634#issuecomment-633689329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org