You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/10/14 08:47:29 UTC
[GitHub] [incubator-kyuubi] pan3793 opened a new issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
pan3793 opened a new issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.
### Describe the proposal
Rework `kyuubi-hive-jdbc` module, remove hive dependencies to make `kyuubi-hive-jdbc` light and maintainable.
### Task list
- #1147
- #1191
- #1203
- #1149
- #1151
- #1152
- #1231
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597
I find some clues after some investigation.
If the module `b` depends on the module `a`, `mvn` will add packaged jar of `a` to its classpath, but IDEA always adds `a/target/classes` instead of jar, that why IDEA always see unrelocated classes.
We can use `build-helper-maven-plugin` to make IDEA see packaged jar in the classpath, but the `a/target/classes` always be included and has a high priority.
```
<!--
This is to ensure references to shaded Hive classes can be resolved in IDEs such as Intellij.
For reference: https://youtrack.jetbrains.com/issue/IDEA-126596
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}.jar</file>
<type>jar</type>
<classifier>optional</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
```
```
/Users/chengpan/.sdkman/candidates/java/8.0.272.fx-zulu/zulu-8.jdk/Contents/Home/bin/java "-javaagent:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=60442:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "
...
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/scala-2.12/test-classes:
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar:
...
"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-929816335
you need to create actual sub-task issues instead of use the id of this ticket everytime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597
I find some clues after investigation.
It's a long-standing bug of IDEA over 9 years, see https://youtrack.jetbrains.com/issue/IDEA-93855.
If the module `b` depends on the module `a`, `mvn` will add packaged jar of `a` to its classpath, but IDEA always adds `a/target/classes` instead of jar, that why IDEA always see unrelocated classes.
We can use `build-helper-maven-plugin` to make IDEA see packaged jar in the classpath, but the `a/target/classes` always be included and has a high priority.
```
<!--
This is to ensure references to shaded Hive classes can be resolved in IDEs such as Intellij.
For reference: https://youtrack.jetbrains.com/issue/IDEA-126596
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}.jar</file>
<type>jar</type>
<classifier>optional</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
```
```
/Users/chengpan/.sdkman/candidates/java/8.0.272.fx-zulu/zulu-8.jdk/Contents/Home/bin/java "-javaagent:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=60442:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "
...
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/scala-2.12/test-classes:
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar:
...
"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943081746
> Put vanilla hive-jdbc jar to co-exist with shaded one is not the right direction.
Got.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943076807
Met two kinds exception, may we need two dependencies.
```
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.jdbc.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-service</artifactId>
<version>${hive.jdbc.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/hive/jdbc/HiveConnection
java.lang.NoClassDefFoundError: org/apache/hive/jdbc/HiveConnection
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/hive/service/cli/HiveSQLException
java.lang.NoClassDefFoundError: org/apache/hive/service/cli/HiveSQLException
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:141)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597
I find some clues after investigation.
It's a long-standing bug of IDEA over 9 years, see https://youtrack.jetbrains.com/issue/IDEA-93855.
If the module `b` depends on the module `a`, `mvn` will add packaged jar of `a` to its classpath, but IDEA always adds `a/target/classes` instead of jar, that why IDEA always see unrelocated classes.
We can use `build-helper-maven-plugin` to make IDEA see packaged jar in the classpath, but the `a/target/classes` always be included and has a high priority.
```
<!--
This is to ensure references to shaded Hive classes can be resolved in IDEs such as Intellij.
For reference: https://youtrack.jetbrains.com/issue/IDEA-126596
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}.jar</file>
<type>jar</type>
<classifier>optional</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
```
```
/Users/chengpan/.sdkman/candidates/java/8.0.272.fx-zulu/zulu-8.jdk/Contents/Home/bin/java "-javaagent:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=60442:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "
...
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/scala-2.12/classes:
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar:
...
"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-929816335
you need to create actual sub-task issues instead of use the id of this ticket everytime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943055968
I can reproduce, really sad news :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943052729
cc @pan3793, still can not found class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 removed a comment on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 removed a comment on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943147547
After some investiagtion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943147547
After some investiagtion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
yaooqinn closed issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-950058334
Close because all sub-tasks have been completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943078770
Put vanilla hive-jdbc jar to co-exist with shaded one is not the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on issue #1131: [Umbrella] Rework kyuubi-hive-jdbc module
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on issue #1131:
URL: https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597
I find some clues after investigation.
If the module `b` depends on the module `a`, `mvn` will add packaged jar of `a` to its classpath, but IDEA always adds `a/target/classes` instead of jar, that why IDEA always see unrelocated classes.
We can use `build-helper-maven-plugin` to make IDEA see packaged jar in the classpath, but the `a/target/classes` always be included and has a high priority.
```
<!--
This is to ensure references to shaded Hive classes can be resolved in IDEs such as Intellij.
For reference: https://youtrack.jetbrains.com/issue/IDEA-126596
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}.jar</file>
<type>jar</type>
<classifier>optional</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
```
```
/Users/chengpan/.sdkman/candidates/java/8.0.272.fx-zulu/zulu-8.jdk/Contents/Home/bin/java "-javaagent:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=60442:
/Users/chengpan/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/212.5284.40/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "
...
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/scala-2.12/test-classes:
/Users/chengpan/Projects/apache-kyuubi/kyuubi-hive-jdbc/target/kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar:
...
"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org