You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/14 21:29:16 UTC
[GitHub] [spark] hvanhovell opened a new pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
hvanhovell opened a new pull request #29752:
URL: https://github.com/apache/spark/pull/29752
### What changes were proposed in this pull request?
This PR refactors the way we propagate the options from the `SparkSession.Builder` to the` SessionState`. This currently done via a mutable map inside the SparkSession. These setting settings are then applied **after** the Session. This is a bit confusing when you expect something to be set when constructing the `SessionState`. This PR passes the options as a constructor parameter to the `SessionStateBuilder` and this will set the options when the configuration is created.
### Why are the changes needed?
It makes it easier to reason about the configurations set in a SessionState than before. We recently had an incident where someone was using `SparkSessionExtensions` to create a planner rule that relied on a conf to be set. While this is in itself probably incorrect usage, it still illustrated this somewhat funky behavior.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692332223
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692335667
**[Test build #128671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128671/testReport)** for PR 29752 at commit [`1543aaf`](https://github.com/apache/spark/commit/1543aaf70a844a8acad0491f1254601236280f70).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692325048
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692393676
**[Test build #128671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128671/testReport)** for PR 29752 at commit [`1543aaf`](https://github.com/apache/spark/commit/1543aaf70a844a8acad0491f1254601236280f70).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692494433
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692332223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692325048
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692389226
+1 I like this idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692394230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692327356
**[Test build #128668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128668/testReport)** for PR 29752 at commit [`73400ed`](https://github.com/apache/spark/commit/73400ed4c21b2383f6ae8567cfc4c6c623d26db9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692327356
**[Test build #128668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128668/testReport)** for PR 29752 at commit [`73400ed`](https://github.com/apache/spark/commit/73400ed4c21b2383f6ae8567cfc4c6c623d26db9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692336104
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692332202
**[Test build #128668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128668/testReport)** for PR 29752 at commit [`73400ed`](https://github.com/apache/spark/commit/73400ed4c21b2383f6ae8567cfc4c6c623d26db9).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HiveSessionStateBuilder(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692394230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692336104
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29752:
URL: https://github.com/apache/spark/pull/29752
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692332228
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128668/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29752: [SPARK-32879][SQL] Refactor SparkSession initial options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29752:
URL: https://github.com/apache/spark/pull/29752#issuecomment-692335667
**[Test build #128671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128671/testReport)** for PR 29752 at commit [`1543aaf`](https://github.com/apache/spark/commit/1543aaf70a844a8acad0491f1254601236280f70).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org