You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2023/08/11 18:50:23 UTC
[GitHub] [spark] juliuszsompolski opened a new pull request, #42457: [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
juliuszsompolski opened a new pull request, #42457:
URL: https://github.com/apache/spark/pull/42457
### What changes were proposed in this pull request?
SparkConnectExecutionManager is a val field in SparkConnectService, so it gets initialized the first time SparkConnectService object is touched. In some scenarios, this can accidentally happen too early, before anything is initialized. SparkConnectExecutionManager wants to read configs from SparkEnv, and when it's not initialized yet it can fail.
Make SparkConnectExecutionManager lazy. Since it is `private[connect]` it should not then be initialized before the service is started.
Raised https://issues.apache.org/jira/browse/SPARK-44779 as a followup to look at the Spark Connect initialization in general.
### Why are the changes needed?
Fix initialization order.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Got one test failure in one specific environment that failed because of wrong initialization order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #42457: [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #42457:
URL: https://github.com/apache/spark/pull/42457#issuecomment-1675222501
@hvanhovell @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42457: [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42457: [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
URL: https://github.com/apache/spark/pull/42457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42457: [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42457:
URL: https://github.com/apache/spark/pull/42457#issuecomment-1675535392
Merged to master and branch-3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org