You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/08/08 20:40:00 UTC
[jira] [Work logged] (HIVE-26458) Add explicit dependency to commons-dbcp2 in hive-exec module
[ https://issues.apache.org/jira/browse/HIVE-26458?focusedWorklogId=799106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-799106 ]
ASF GitHub Bot logged work on HIVE-26458:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Aug/22 20:39
Start Date: 08/Aug/22 20:39
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request, #3510:
URL: https://github.com/apache/hive/pull/3510
### Why are the changes needed?
See HIVE-26458
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Existing tests.
- Cherry-picking the change in internal Hive branches and running `TestMiniLLapLocalCliDriver` with tests including external JDBC tables
Issue Time Tracking
-------------------
Worklog Id: (was: 799106)
Remaining Estimate: 0h
Time Spent: 10m
> Add explicit dependency to commons-dbcp2 in hive-exec module
> ------------------------------------------------------------
>
> Key: HIVE-26458
> URL: https://issues.apache.org/jira/browse/HIVE-26458
> Project: Hive
> Issue Type: Task
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive CBO relies on Calcite so there is a direct dependency towards Calcite in hive-exec module. On its turn, Calcite needs commons-dbcp2 dependency in order to compile and run properly:
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/build.gradle.kts#L69
> In particular the dependency is necessary in order to use the JDBC adapter and some of its usages are shown below:
> * https://github.com/apache/calcite/blob/257c81b5cac35e29598a246463356fea7e0b0336/core/src/main/java/org/apache/calcite/adapter/jdbc/JdbcUtils.java#L29
> * https://github.com/apache/calcite/blob/257c81b5cac35e29598a246463356fea7e0b0336/core/src/main/java/org/apache/calcite/adapter/jdbc/JdbcUtils.java#L262
> However, due to the [shading of Calcite|https://github.com/apache/hive/blob/778c838317c952dcd273fd6c7a51491746a1d807/ql/pom.xml#L1075] inside hive-exec module all the transitive dependencies coming from Calcite must be defined explicitly otherwise they will not make it to the classpath.
> At the moment this does not pose a problem in master since {{commons-dbcp2}} dependency comes transitively from other modules. But in certain Hive branches with slightly different dependencies between modules we have seen failures like the one shown below:
> {noformat}
> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/commons/dbcp2/BasicDataSource
> at org.apache.calcite.adapter.jdbc.JdbcUtils$DataSourcePool.<init>(JdbcUtils.java:213)
> at org.apache.calcite.adapter.jdbc.JdbcUtils$DataSourcePool.<clinit>(JdbcUtils.java:210)
> at org.apache.calcite.adapter.jdbc.JdbcSchema.dataSource(JdbcSchema.java:207)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3331)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5324)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1815)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1750)
> at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.plan(CalcitePlanner.java:1411)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:588)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13071)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:312)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:231)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:867)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837)
> at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:178)
> at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:173)
> at org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:151)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:111)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
> at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
> Caused by: java.lang.NoClassDefFoundError: org/apache/commons/dbcp2/BasicDataSource
> ... 71 more
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.dbcp2.BasicDataSource
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 71 more
> {noformat}
> Without shading there wouldn't be a need to have an explicit dependency to dbcp2 but the way it is currently done it is neccessary to have this. This is the same reason of why janino and other transitive deps from Calcite are declared explicitly in hive-exec module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)