You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/29 08:26:15 UTC
[GitHub] [hive] ayushtkn opened a new pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
ayushtkn opened a new pull request #1187:
URL: https://github.com/apache/hive/pull/1187
https://issues.apache.org/jira/browse/HIVE-23772
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-652211265
> Ideally we can remove the original ones for itests??
I think that will not make the problem go away - if someone adds a calcite-core to the classpath this could happen in production setup as well...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651580913
iirc the point where the switch to the "non-shaded calcite-core" happens is when the jdbc driver is loaded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651325059
yeah; now I remember - I've also reached these avatica exceptions; the story behind them is:
* jdbc driver is used to open the connection; it passes some things as arguments to the jdbc driver
* the typefactory is "org.calcite...""
* the expected type factory is "shaded.org.calcite"
so it fails with an exception...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn edited a comment on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn edited a comment on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651592714
The assertion error is due to the fact both shaded and non shaded jars are there in classpath.
``` Caused by: java.lang.AssertionError
at org.apache.calcite.avatica.AvaticaUtils.instantiatePlugin(AvaticaUtils.java:229) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$5.apply(ConnectionConfigImpl.java:392) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.get_(ConnectionConfigImpl.java:173) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:296) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:282) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.config.CalciteConnectionConfigImpl.typeSystem(CalciteConnectionConfigImpl.java:155) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:127) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:115) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) ~[avatica-1.12.0.jar:1.12.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1555) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:540) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
```
Post ``at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]`` rather than coming back to ``hive-exec`` it goes back to the non shaded jars ``calcite-core``
The driver gets registered with original jar only so it executes call to the non-shaded ones.
Ideally we can remove the original ones for itests??
Removing that takes away that error, but driver registration is done as ``HiveDriver`` and while getting connection it wants a `jdbc:calcite` hard-coded in `Framework` class so it gets a:
``` Caused by: java.sql.SQLException: No suitable driver found for jdbc:calcite:
at java.sql.DriverManager.getConnection(DriverManager.java:689) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
```
Is there are configuration or workaround for the driver-connection. If you have any idea, Let me know
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651363049
The values passed are from :
https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/jdbc/CalciteConnectionImpl.java#L127
The type factory must be null since it is the else part.
The trace says its L127 -
`at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:127`
and both argument ideally should be from `calcite-core` hence should have got relocated, they are further tweaked at `calcite-avatica` only, may be some dependency which uses calcite-core is called, Need to find, For that need to clone and get the dependency tree and stuff, or debug where the signature got changed from shaded to non shaded.
Will find time to dig in more may be this or next weekend for this. Do let me know if you have any further pointers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651055461
Have updated itest as well, which caused the compilation to fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1187:
URL: https://github.com/apache/hive/pull/1187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-688557522
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651592714
The assertion error is due to the fact both shaded and non shaded jars are there in classpath.
``` Caused by: java.lang.AssertionError
at org.apache.calcite.avatica.AvaticaUtils.instantiatePlugin(AvaticaUtils.java:229) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$5.apply(ConnectionConfigImpl.java:392) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.get_(ConnectionConfigImpl.java:173) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:296) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:282) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.config.CalciteConnectionConfigImpl.typeSystem(CalciteConnectionConfigImpl.java:155) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:127) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:115) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) ~[avatica-1.12.0.jar:1.12.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1555) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:540) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
```
Post ``at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]`` rather than coming back to ``hive-exec`` it goes back to the non shaded jars ``calcite-core``
The driver gets registered with original jar only so it executes call to the non-shaded ones.
Ideally we can remove the original ones for itests??
Removing that takes away that error, but driver registration is done as ``HiveDriver`` and while getting connection it wants a `jdbc:calcite` hard-coded in `Framework` class so it gets a:
``` Caused by: java.sql.SQLException: No suitable driver found for jdbc:calcite:
at java.sql.DriverManager.getConnection(DriverManager.java:689) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]```
Is there are configuration or workaround for the driver-connection. If you have any idea, Let me know
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn edited a comment on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn edited a comment on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651363049
The values passed are from :
https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/jdbc/CalciteConnectionImpl.java#L127
The type factory must be null since it is the else part.
The trace says its L127 -
`at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:127`
and both argument ideally should be from `calcite-core` hence should have got relocated, they are further tweaked at `calcite-avatica` only, may be some dependency which uses calcite-core is called, Need to find, For that need to clone and get the dependency tree and stuff, or debug where the signature got changed from shaded to non shaded.
Will find time to dig in more may be this or next weekend for this. Need to see Calcite working as well, Which I have no idea. Do let me know if you have any further pointers. :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn edited a comment on pull request #1187: HIVE-23772. Reallocate calcite-core to prevent NoSuchFiledError.
Posted by GitBox <gi...@apache.org>.
ayushtkn edited a comment on pull request #1187:
URL: https://github.com/apache/hive/pull/1187#issuecomment-651592714
The assertion error is due to the fact both shaded and non shaded jars are there in classpath.
``` Caused by: java.lang.AssertionError
at org.apache.calcite.avatica.AvaticaUtils.instantiatePlugin(AvaticaUtils.java:229) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$5.apply(ConnectionConfigImpl.java:392) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.get_(ConnectionConfigImpl.java:173) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:296) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:282) ~[avatica-1.12.0.jar:1.12.0]
at org.apache.calcite.config.CalciteConnectionConfigImpl.typeSystem(CalciteConnectionConfigImpl.java:155) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:127) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:115) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) ~[avatica-1.12.0.jar:1.12.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1555) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:540) ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
```
Post ``at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_181]`` rather than coming back to ``hive-exec`` it goes back to the non shaded jars ``calcite-core``
The driver gets registered with original jar only so it executes call to the non-shaded ones.
Ideally we can remove the original ones for itests??
Removing that takes away that error, but driver registration is done as ``HiveDriver`` and while getting connection it wants a `jdbc:calcite` hard-coded in `Framework` class so it gets a:
```
Caused by: java.sql.SQLException: No suitable driver found for jdbc:calcite:
at java.sql.DriverManager.getConnection(DriverManager.java:689) ~[?:1.8.0_181]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_181]
at org.apache.hive.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:175) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
```
Is there are configuration or workaround for the driver-connection. If you have any idea, Let me know
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org