You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/20 18:07:52 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27653: [MINOR][SQL] Add
a comment for `removedSQLConfigs`
MaxGekk opened a new pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653
### What changes were proposed in this pull request?
In the PR, I propose to explain in the description of `removedSQLConfigs` when removed SQL configs should NOT be placed to the map.
### Why are the changes needed?
To make the cases when SQL configs should be added to `removedSQLConfigs` more clear. Recently, `spark.sql.variable.substitute.depth` was removed from the map by #27646 because it contradicts to the condition described by the PR.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
By `./dev/scalastyle`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589233725
**[Test build #118728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118728/testReport)** for PR 27653 at commit [`da9873c`](https://github.com/apache/spark/commit/da9873c8e33aa342991898ae2cc2007af1cf8bd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589437863
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589234212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23479/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589234212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23479/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382179037
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,14 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
+ * the behavior. Any removed config of `removedSQLConfigs` which set to a non-default
+ * value raises an exception propagated to user's app. For example, the SQL config
+ * `spark.sql.variable.substitute.depth` was deprecated since Spark 2.1 and not
+ * used in Spark 2.4, so, it should not be placed here because any of its values
+ * does not impact on the behavior of Spark 3.0. And this makes migrations to new
Review comment:
Shall we describe in a simpler and positive words like `because the config is no-op technically` instead of `because any of its values does not impact on the behavior of Spark`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279938
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23483/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589895048
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118805/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589895044
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279930
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589895542
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589895048
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118805/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589823519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23555/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589823519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23555/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589234195
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279930
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589390010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118728/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27653: [MINOR][SQL] Add
a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589825659
**[Test build #118805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118805/testReport)** for PR 27653 at commit [`43ea1a6`](https://github.com/apache/spark/commit/43ea1a6399f8711c0cf87cc959166e7d96e87592).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589241325
Thanks, @MaxGekk .
cc @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589437283
**[Test build #118732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118732/testReport)** for PR 27653 at commit [`c48364a`](https://github.com/apache/spark/commit/c48364a5ae2088cd3d2d257dfd68db750fac676a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279258
**[Test build #118732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118732/testReport)** for PR 27653 at commit [`c48364a`](https://github.com/apache/spark/commit/c48364a5ae2088cd3d2d257dfd68db750fac676a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589389206
**[Test build #118728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118728/testReport)** for PR 27653 at commit [`da9873c`](https://github.com/apache/spark/commit/da9873c8e33aa342991898ae2cc2007af1cf8bd7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589234195
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382410179
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,13 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
+ * the behavior. Any removed config of `removedSQLConfigs` which set to a non-default
+ * value raises an exception propagated to user's app. For example, the SQL config
+ * `spark.sql.variable.substitute.depth` was deprecated since Spark 2.1 and not
+ * used in Spark 2.4, so, it should not be placed here because the config
Review comment:
actually, we should've put in it remove configs in 2.4...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589894766
**[Test build #118805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118805/testReport)** for PR 27653 at commit [`43ea1a6`](https://github.com/apache/spark/commit/43ea1a6399f8711c0cf87cc959166e7d96e87592).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589823514
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589895044
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382357860
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,13 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
Review comment:
Another suggestion is like:
> Adds removed SQL configurations here only when the configuration affects behaviours. For example, `spark.sql.variable.substitute.depth` configuration was not added as it virtually became no-op later. By this, it makes migrations to new Spark versions painless.
I think we can write this simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27653: [MINOR][SQL] Add
a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589233725
**[Test build #118728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118728/testReport)** for PR 27653 at commit [`da9873c`](https://github.com/apache/spark/commit/da9873c8e33aa342991898ae2cc2007af1cf8bd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589390001
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589437871
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118732/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589823514
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382356996
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,14 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
+ * the behavior. Any removed config of `removedSQLConfigs` which set to a non-default
+ * value raises an exception propagated to user's app. For example, the SQL config
+ * `spark.sql.variable.substitute.depth` was deprecated since Spark 2.1 and not
+ * used in Spark 2.4, so, it should not be placed here because any of its values
+ * does not impact on the behavior of Spark 3.0. And this makes migrations to new
Review comment:
Yeah, I was about to point this out. Let's make the description shorter and simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589390010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118728/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382357860
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,13 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
Review comment:
Another suggestion is like:
> Adds removed SQL configurations here only when the configuration affects behaviours. For example, `spark.sql.variable.substitute.depth` configuration was not added as it virtually became no-op later.
I think we can write this simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589825659
**[Test build #118805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118805/testReport)** for PR 27653 at commit [`43ea1a6`](https://github.com/apache/spark/commit/43ea1a6399f8711c0cf87cc959166e7d96e87592).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589390001
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589437871
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118732/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382357860
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,13 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
Review comment:
Another suggestion is like:
> Adds removed SQL configurations here only when the configuration affects behaviours. For example, `spark.sql.variable.substitute.depth` configuration was not added as it virtually become no-op.
I think we can write this simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279938
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23483/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382357860
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,13 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
Review comment:
Another suggestion is like (feel free to rephrase):
> Adds removed SQL configurations here only when the configuration affects behaviours. For example, `spark.sql.variable.substitute.depth` configuration was not added as it virtually became no-op later. By this, it makes migrations to new Spark versions painless.
I think we can write this simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27653:
[MINOR][SQL] Add a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#discussion_r382356996
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2254,6 +2254,14 @@ object SQLConf {
* The map contains info about removed SQL configs. Keys are SQL config names,
* map values contain extra information like the version in which the config was removed,
* config's default value and a comment.
+ *
+ * Removed SQL config should be added to the map iff its non-default value changed
+ * the behavior. Any removed config of `removedSQLConfigs` which set to a non-default
+ * value raises an exception propagated to user's app. For example, the SQL config
+ * `spark.sql.variable.substitute.depth` was deprecated since Spark 2.1 and not
+ * used in Spark 2.4, so, it should not be placed here because any of its values
+ * does not impact on the behavior of Spark 3.0. And this makes migrations to new
Review comment:
Yeah, I was about to point this out. Let's make the description shorter and simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27653: [MINOR][SQL] Add
a comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589279258
**[Test build #118732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118732/testReport)** for PR 27653 at commit [`c48364a`](https://github.com/apache/spark/commit/c48364a5ae2088cd3d2d257dfd68db750fac676a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a
comment for `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27653: [MINOR][SQL] Add a comment for `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27653#issuecomment-589437863
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org