You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2017/03/14 15:50:41 UTC
[jira] [Commented] (CALCITE-1694) Clash in protobuf versions
between Avatica and Hadoop
[ https://issues.apache.org/jira/browse/CALCITE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15924448#comment-15924448 ]
Josh Elser commented on CALCITE-1694:
-------------------------------------
bq. But the real solution, I think, is too reduce the dependencies of ConnectionPropertiesImpl. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the fromProto(Common.ConnectionProperties) method.
Agreed -- it's been on my back-burner to separate out the "transports" to their own respective modules from "avatica-core".
However.. {{Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3}}, this looks like the non-shaded Avatica artifacts are being used (again). We ran into this a few months back. Let me look at the calcite-pig dependency graph.
> Clash in protobuf versions between Avatica and Hadoop
> -----------------------------------------------------
>
> Key: CALCITE-1694
> URL: https://issues.apache.org/jira/browse/CALCITE-1694
> Project: Calcite
> Issue Type: Bug
> Components: avatica, pig-adapter
> Reporter: Julian Hyde
>
> When running Pig adapter tests, we get the following stack:
> {noformat}
> java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.calcite.avatica.ConnectionPropertiesImpl.<clinit>(ConnectionPropertiesImpl.java:37)
> at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
> at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
> at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
> at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
> at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
> at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
> at org.apache.calcite.test.PigAdapterTest.testImplWithGroupByMultipleFields(PigAdapterTest.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 58 more
> {noformat}
> followed by
> {noformat}
> java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.avatica.ConnectionPropertiesImpl
> at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
> at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
> at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
> at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
> at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
> at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
> at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
> at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
> at org.apache.calcite.test.PigAdapterTest.testImplWithMultipleFilters(PigAdapterTest.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> {noformat}
> If you explicitly load {{ConnectionPropertiesImpl}} by adding the line
> {code}
> Util.discard(org.apache.calcite.avatica.ConnectionPropertiesImpl.class);
> {code}
> to {{PigSchemaFactory}} the error goes away.
> I think the problem might be that the Pig adapter pulls in Pig, which pulls in Hadoop, which depends on protobuf 2, whereas Avatica's {{ConnectionPropertiesImpl}} has some members that depend on protobuf 3.
> We can work around by having the Pig adapter load Calcite's {{MetaImpl}} (thereby avoiding an explicit dependency of the Pig adapter on Avatica).
> But the real solution, I think, is too reduce the dependencies of {{ConnectionPropertiesImpl}}. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the {{fromProto(Common.ConnectionProperties)}} method.
> [~elserj], What do you think?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)