You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/29 13:46:49 UTC
[GitHub] [spark] kevincmchen opened a new pull request #34742: using the same jar class loader in HiveSessionResourceLoader and HiveClient
kevincmchen opened a new pull request #34742:
URL: https://github.com/apache/spark/pull/34742
### What changes were proposed in this pull request?
this PR
### Why are the changes needed?
when using livy to execute sql statements that will use the udf jars located in lakefs, a inner filesystem in Tencent Cloud DLC. it will threw the following exceptions:
`java.lang.LinkageError: loader constraint violation: loader (instance of sun/misc/Launcher$AppClassLoader) previously initiated loading for a different type with name`
Maybe it happens in some other filesystem.
This PR fix the bug by using the same jar class loader in `HiveSessionResourceLoader` and `HiveClient`
### Why are the changes needed?
fix the bug that different class loader to load the same class.
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
existing testsuites
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen closed pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen closed pull request #34742:
URL: https://github.com/apache/spark/pull/34742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-983645625
> Thank you for your explanation. If you are sure it will not affect the old case, I think this change is OK
ok, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen removed a comment on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen removed a comment on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-982253151
> filing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen closed pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen closed pull request #34742:
URL: https://github.com/apache/spark/pull/34742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34742: set the setContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-982181025
@kevincmchen mind filing a JIRA? see also https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-982257963
> @kevincmchen mind filing a JIRA? see also https://spark.apache.org/contributing.html
@HyukjinKwon ok, i have created a [issue](https://issues.apache.org/jira/browse/SPARK-37486) about the PR and you can get the details about the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-983262803
Thank you for your explanation. If you are sure it will not affect the old case, I think this change is OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-982253151
> filing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-983650416
@HyukjinKwon Could you please review the code and merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34742: set the setContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-981678108
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen edited a comment on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen edited a comment on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-983257922
> I'm not sure if there is a counterexample that needs to call `client.addJar` first
I don't think `supper.addJars` and `client.addJars` are strongly dependent.
the reason why i call the supper.addJars first is to set the ContextClassLoader of current thread, to avoid that different class loaders load the same class in HiveClient。 @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kevincmchen commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
kevincmchen commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-983257922
>
> I'm not sure if there is a counterexample that needs to call `client.addJar` first
I don't think `supper.addJars` and `client.addJars` are strongly dependent.
the reason why i call the supper.addJars first is to set the ContextClassLoader of current thread, to avoid that different class loaders load the same class in HiveClient。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34742: [SPARK-37486][SQL][HIVE] set the ContextClassLoader before using the `addJars` in `HiveClient`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34742:
URL: https://github.com/apache/spark/pull/34742#issuecomment-982691202
I'm not sure if there is a counterexample that needs to call `client.addJar` first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org