You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "zhengchenyu (Jira)" <ji...@apache.org> on 2020/08/11 11:03:00 UTC
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should
shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175452#comment-17175452 ]
zhengchenyu edited comment on HIVE-22126 at 8/11/20, 11:02 AM:
---------------------------------------------------------------
When I run the program, I found an AbstractMethodError. I think HiveAggregateFactoryImpl's createAggregate doesn't implement AggregateFactory, I think com.google.common.collect.ImmutableList would be relocation to org.apache.hive.com.google.common.collect.ImmutableList, so throw AbstractMethodError.
Notes: for shade all guava jar, I remove guava lib from lib dir and use maven-shade-plugin in main pom.xml to shade all guava.
Error stack are below:
{code:java}
2020-08-11T18:26:51,434 ERROR [0ae58217-4908-4697-a9e7-a57f279a22a0 main] parse.CalcitePlanner: CBO failed, skipping CBO.
java.lang.RuntimeException: java.lang.AbstractMethodError: org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.rethrowCalciteException(CalcitePlanner.java:1539) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1417) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1430) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:450) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12164) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) ~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) ~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) ~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) ~[hive-cli-3.1.2.jar:3.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.hadoop.util.RunJar.run(RunJar.java:323) ~[hadoop-common-3.2.1.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:236) ~[hadoop-common-3.2.1.jar:?]
Caused by: java.lang.AbstractMethodError: org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:1267) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:886) ~[calcite-core-1.16.0.jar:1.16.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimChild(HiveRelFieldTrimmer.java:125) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:377) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimFields(HiveRelFieldTrimmer.java:315) ~[hive-exec-3.1.2.jar:3.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:162) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1692) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111) ~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1414) ~[hive-exec-3.1.2.jar:3.1.2]
... 23 more
{code}
was (Author: zhengchenyu):
When I run the program, I found an AbstractMethodError. I think HiveAggregateFactoryImpl's createAggregate doesn't implement AggregateFactory, I think com.google.common.collect.ImmutableList would be relocation to org.apache.hive.com.google.common.collect.ImmutableList, so throw AbstractMethodError.
Notes: for shade all guava jar, I remove guava lib from lib dir and use maven-shade-plugin in main pom.xml to shade all guava.
Error stack are below:
{code}
2020-08-11T18:26:51,434 ERROR [0ae58217-4908-4697-a9e7-a57f279a22a0 main] parse.CalcitePlanner: CBO failed, skipping CBO.2020-08-11T18:26:51,434 ERROR [0ae58217-4908-4697-a9e7-a57f279a22a0 main] parse.CalcitePlanner: CBO failed, skipping CBO.java.lang.RuntimeException: java.lang.AbstractMethodError: org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode; at org.apache.hadoop.hive.ql.parse.CalcitePlanner.rethrowCalciteException(CalcitePlanner.java:1539) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1417) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1430) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:450) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12164) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) ~[hive-cli-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) ~[hive-cli-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) ~[hive-cli-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) ~[hive-cli-3.1.2.jar:3.1.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241] at org.apache.hadoop.util.RunJar.run(RunJar.java:323) ~[hadoop-common-3.2.1.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:236) ~[hadoop-common-3.2.1.jar:?]Caused by: java.lang.AbstractMethodError: org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode; at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:1267) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:886) ~[calcite-core-1.16.0.jar:1.16.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimChild(HiveRelFieldTrimmer.java:125) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:377) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimFields(HiveRelFieldTrimmer.java:315) ~[hive-exec-3.1.2.jar:3.1.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_241] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241] at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:162) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1692) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111) ~[calcite-core-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1414) ~[hive-exec-3.1.2.jar:3.1.2] ... 23 more
{code}
> hive-exec packaging should shade guava
> --------------------------------------
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Eugene Chung
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a problems for downstream clients of hive which have hive-exec.jar in their classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)