You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/20 11:22:42 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
MaxGekk opened a new pull request #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646
### What changes were proposed in this pull request?
Exclude the SQL config `spark.sql.variable.substitute.depth` from `SQLConf.removedSQLConfigs`
### Why are the changes needed?
By the #27169, the config was placed to `SQLConf.removedSQLConfigs`. And as a consequence of that when an user set it non-default value (1 for example), he/she will get an exception. It is acceptable for SQL configs that could impact on the behavior but not for this particular config. Raising of such exception will just make migration to Spark 3.0 more difficult.
### Does this PR introduce any user-facing change?
Yes, before the changes users get an exception when he/she set `spark.sql.variable.substitute.depth` to a value different from `40`.
### How was this patch tested?
Run `spark.conf.set("spark.sql.variable.substitute.depth", 1)` in `spark-shell`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589176588
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973690
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23467/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589145803
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118716/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589145792
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973285
**[Test build #118716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118716/testReport)** for PR 27646 at commit [`0eac19e`](https://github.com/apache/spark/commit/0eac19eaf2b0d269880d79439a6087989d1b6895).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589459043
Okay LGTM too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973690
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23467/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589144262
**[Test build #118716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118716/testReport)** for PR 27646 at commit [`0eac19e`](https://github.com/apache/spark/commit/0eac19eaf2b0d269880d79439a6087989d1b6895).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588980508
This config was already not in use when we remove it #27169. If it has no effect, we don't need to throw exception when it is set. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973285
**[Test build #118716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118716/testReport)** for PR 27646 at commit [`0eac19e`](https://github.com/apache/spark/commit/0eac19eaf2b0d269880d79439a6087989d1b6895).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589237261
So, the idea is simply ignoring this config because it's no-op?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-588973680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#discussion_r382168771
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2263,8 +2263,6 @@ object SQLConf {
"It was removed to prevent loosing of users data for non-default value."),
RemovedConfig("spark.sql.legacy.compareDateTimestampInTimestamp", "3.0.0", "true",
"It was removed to prevent errors like SPARK-23549 for non-default value."),
- RemovedConfig("spark.sql.variable.substitute.depth", "3.0.0", "40",
Review comment:
Here is the PR https://github.com/apache/spark/pull/27653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589237261
So, the idea is simply ignoring this config because it's no-op?
+1, late LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589145792
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27646: [SPARK-30892][SQL]
Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#issuecomment-589145803
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118716/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27646:
[SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from
`removedSQLConfigs`
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #27646: [SPARK-30892][SQL] Exclude `spark.sql.variable.substitute.depth` from `removedSQLConfigs`
URL: https://github.com/apache/spark/pull/27646#discussion_r382121967
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2263,8 +2263,6 @@ object SQLConf {
"It was removed to prevent loosing of users data for non-default value."),
RemovedConfig("spark.sql.legacy.compareDateTimestampInTimestamp", "3.0.0", "true",
"It was removed to prevent errors like SPARK-23549 for non-default value."),
- RemovedConfig("spark.sql.variable.substitute.depth", "3.0.0", "40",
Review comment:
Could we add the description in line 2255 and explain which removed SQLConfigs should NOT be added to removedSQLConfigs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org