You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2024/01/03 09:57:59 UTC
[PR] [SPARK-46566][SQL] Session level config was not loaded when isolation is enable. [spark]
zhengchenyu opened a new pull request, #44572:
URL: https://github.com/apache/spark/pull/44572
### What changes were proposed in this pull request?
Load session level config.
### Why are the changes needed?
I found session level hive config is missing when we use isolation client classloader for specified hive metastore version.
For more information see: https://issues.apache.org/jira/browse/SPARK-46566
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
test in our prod yarn cluster.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46566][SQL] Session level config was not loaded when isolation is enable. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #44572: [SPARK-46566][SQL] Session level config was not loaded when isolation is enable.
URL: https://github.com/apache/spark/pull/44572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46566][SQL] Session level config was not loaded when isolation is enable. [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #44572:
URL: https://github.com/apache/spark/pull/44572#issuecomment-1875151150
Each session should use its own hiveconfig. HiveSessionImpl::acquire sets the SessionSate of the current session to the threadlocal variable. Theoretically we can use SessionState.get directly and then get our own hiveconfig from it. However, since IsolatedClientLoader will reload the SessionState class, and SessionState::tss is a static variable, SessionState::tss will be reinitialized, so SessionState is lost.
Therefore, I introduced HiveContext::sessionHiveConfBuffer to store hiveConfig. Because HiveContext is a sharedClass, the class will not be reloaded and the sessionHiveConfBuffer will not be reinitialized.
Why introduce bytebuffer instead of hiveConfig directly? Because the version of HiveConfig builtin spark may be different from the version of HiveConfig loaded by IsolatedClientLoader, and an error will be reported when running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46566][SQL] Session level config was not loaded when isolation is enable. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44572:
URL: https://github.com/apache/spark/pull/44572#issuecomment-2052721053
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org