You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/07/08 19:31:39 UTC
[GitHub] [hive] viirya opened a new pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
viirya opened a new pull request #2459:
URL: https://github.com/apache/hive/pull/2459
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Trying to relocate dependencies which could conflict Spark.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
When we want to use shaded version of hive-exec (i.e., w/o classifier), more dependencies conflict with Spark. We need to relocate these dependencies too.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
If previously downstream projects rely on included dependencies in shaded release, they might need to explicitly include these dependencies after the relocation here.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r678304214
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
don't use the core artifact - that's just bad!
what are you trying to achieve here with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-879417843
@viirya there are still more errors:
```
java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/logging/LogFactory
at org.apache.hadoop.hive.conf.valcoersion.JavaIOTmpdirVariableCoercion.<clinit>(JavaIOTmpdirVariableCoercion.java:35)
at org.apache.hadoop.hive.conf.SystemVariables.<clinit>(SystemVariables.java:37)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:3445)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:3409)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:436)
at org.apache.hadoop.hive.druid.serde.DruidSerDe.initialize(DruidSerDe.java:113)
at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at org.apache.hadoop.hive.druid.TestDruidSerDe.testDruidDeserializer(TestDruidSerDe.java:510)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
```
```
Caused by: java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/codec/language/Soundex
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFSoundex.<init>(GenericUDFSoundex.java:49)
... 29 more
```
```
Caused by: java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/codec/binary/Base64
at org.apache.hadoop.hive.serde2.lazy.LazyBinary.decodeIfNeeded(LazyBinary.java:58)
at org.apache.hadoop.hive.serde2.lazy.LazyBinary.init(LazyBinary.java:50)
at org.apache.hadoop.hive.serde2.lazy.LazyStruct.uncheckedGetField(LazyStruct.java:226)
at org.apache.hadoop.hive.serde2.lazy.LazyStruct.getField(LazyStruct.java:202)
at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:128)
at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:157)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:95)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:438)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:430)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881728274
I see. Yea this is very weird. I suspect there are something else that caused it to fail, for instance, when initializing some of the enum items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881720413
The quoted test failure from "Testing / split-20 / Archive / testCommitCreateTablePlusCommitDropTableWithoutPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler" in http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2459/11/tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r709911738
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
note: on branch2 guava is most likely not properly shaded away HIVE-22126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881720606
Yea, I'm trying to deal with `org.apache.hive.org.apache.commons.logging.LogFactory`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-919772056
Close this as we are taking a different direction to shade dependencies of Hive at Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r678307084
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
branch-2.3 ?
please note that changes should land on master first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-879269084
Thanks @viirya for the update! there are still some failures that are related, such as
1.
```
java.lang.NoSuchMethodError: org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol.newReflectiveBlockingService(Lorg/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos$LlapDaemonProtocol$BlockingInterface;)Lcom/google/protobuf/BlockingService;
at org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.serviceStart(LlapProtocolServerImpl.java:142)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStart(LlapDaemon.java:405)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.hive.llap.daemon.MiniLlapCluster.serviceStart(MiniLlapCluster.java:177)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.hive.llap.LlapItUtils.startAndGetMiniLlapCluster(LlapItUtils.java:78)
at org.apache.hadoop.hive.ql.QTestUtil.setupMiniCluster(QTestUtil.java:656)
at org.apache.hadoop.hive.ql.QTestUtil.<init>(QTestUtil.java:584)
at org.apache.hadoop.hive.cli.control.CoreCliDriver$1.invokeInternal(CoreCliDriver.java:65)
at org.apache.hadoop.hive.cli.control.CoreCliDriver$1.invokeInternal(CoreCliDriver.java:60)
at org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33)
at org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:67)
at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:71)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
```
2.
```
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(Ljava/util/Properties;Ljava/util/List;Ljava/util/List;Ljava/lang/String;)Lorg/apache/avro/Schema;
at org.apache.hive.hcatalog.mapreduce.SpecialCases.addSpecialCasesParametersToOutputJobProperties(SpecialCases.java:125)
at org.apache.hive.hcatalog.mapreduce.FosterStorageHandler.configureOutputJobProperties(FosterStorageHandler.java:217)
at org.apache.hive.hcatalog.common.HCatUtil.configureOutputStorageHandler(HCatUtil.java:498)
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:198)
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
at org.apache.hive.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:191)
at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:57)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1851)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1527)
at org.apache.pig.PigServer.execute(PigServer.java:1440)
at org.apache.pig.PigServer.executeBatch(PigServer.java:488)
at org.apache.pig.PigServer.executeBatch(PigServer.java:471)
at org.apache.hive.hcatalog.pig.TestHCatStorerMulti.testStoreBasicTable(TestHCatStorerMulti.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r710388364
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
we could relocate/shade away those deps to make it possible for other projects to use the normal artifact - seems like there is a very good list in the trino project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r696816642
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
Yea I think so since this PR is trying to shade things from the `hive-exec-core` I believe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r709908764
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
we should fix the issues with using the normal `hive-exec` artifact if there is any - loading the core jar could cause troubles...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r710369202
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
@kgyrtkirk Guava is shaded in branch-2.3 via https://issues.apache.org/jira/browse/HIVE-23980. The issue is, in order for Spark to use shaded `hive-exec`, Hive will need to relocate more classes and at the same time making sure it won't break other modules (for instance, if the shaded class appears in certain API and another module imported the unshaded version of the class by itself).
Currently we've abandoned this approach and decided to shade the `hive-exec-core` within Spark itself, following similar approach in Trino (see https://github.com/trinodb/trino-hive-apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r695185072
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
As more dependencies are relocated here, some modules if they depends on non-core artifact, will cause class not found error...
The motivation is because we want to use shaded version of hive-exec (i.e., w/o classifier) in Spark to make sure it doesn't conflict guava version there. But there are more dependencies conflict with Spark. We need to relocate these dependencies too..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao edited a comment on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881716282
I'll take a look @viirya - where did you find this error? do you have a link?
I'm seeing these errors in the last run:
```
Caused by: java.lang.ClassNotFoundException: org.apache.hive.org.apache.commons.codec.language.Soundex
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 32 more
```
```
Caused by: java.lang.ClassNotFoundException: org.apache.hive.org.apache.commons.logging.LogFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 35 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya closed pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #2459:
URL: https://github.com/apache/hive/pull/2459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-879231076
cc @sunchao Most test failures are passed now. Please check if current CI is okay? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881716282
I'll take a look @viirya - where did you find this error? do you have a link?
I'm seeing this error in the last run:
```
Caused by: java.lang.ClassNotFoundException: org.apache.hive.org.apache.commons.codec.language.Soundex
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 32 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on a change in pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #2459:
URL: https://github.com/apache/hive/pull/2459#discussion_r695352477
##########
File path: llap-server/pom.xml
##########
@@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
+ <classifier>core</classifier>
Review comment:
@sunchao do we need to have similar change on master first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-879374726
Thanks @sunchao. Seems caused by relocating avro, protobuf. Reverted the two relocations and see what's going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] viirya commented on pull request #2459: HIVE-25317: Relocate dependencies in shaded hive-exec module
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #2459:
URL: https://github.com/apache/hive/pull/2459#issuecomment-881712574
I'm not sure why this happens, or if it is caused by this change. `HiveConf` is not from hive-exec (ql), and this doesn't relocate any Hive internal class (we shoudn't do this too). cc @sunchao
```
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf$ConfVars
at org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateTablePlusCommitDropTableWithoutPurge(TestDruidStorageHandler.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org