You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2023/03/29 08:57:31 UTC
[GitHub] [incubator-paimon] tsreaper opened a new pull request, #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
tsreaper opened a new pull request, #764:
URL: https://github.com/apache/incubator-paimon/pull/764
### Purpose
This PR solves #660 .
Currently, some classes from Hive 2.3 are bundled into the Paimon Hive catalog. However, this can lead to conflicts when users are using a different version of Hive.
Furthermore, the current Hive catalog only considers Hive 2.x, but the parameters of RetryingMetaStoreClient#getProxy differ between Hive 2.x and Hive 3.x. This can result in a NoSuchMethod exception for users of Hive 3.x.
Once this pull request is merged, the Paimon Hive catalog will be small and independent enough to be bundled into the distributed jars for all supported engines.
### Tests
* Hive23CatalogITCase
* Hive31CatalogITCase
### API and Format
N/A
### Documentation
This pull request may only affect Flink users. Once it is merged, the Paimon Hive catalog for Flink will rely on the Flink Hive connector bundled jar, as indicated in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] tsreaper commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152731412
##########
paimon-e2e-tests/pom.xml:
##########
@@ -34,6 +34,7 @@ under the License.
<properties>
<flink.shaded.hadoop.version>2.8.3-10.0</flink.shaded.hadoop.version>
<flink.sql.connector.kafka>flink-sql-connector-kafka</flink.sql.connector.kafka>
+ <flink.sql.connector.hive>flink-sql-connector-hive-2.3.9_${scala.binary.version}</flink.sql.connector.hive>
Review Comment:
I'd love to. However currently I can't find any Hive 3.x docker image. Let's create an issue for adding Hive 3.x e2e tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] SteNicholas merged pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas merged PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] SteNicholas commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152686556
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -470,17 +472,53 @@ private Lock lock(Identifier identifier) {
return Lock.fromCatalog(lock, identifier);
}
+ private static final List<Class<?>[]> GET_PROXY_PARAMS =
+ Arrays.asList(
+ // for hive 2.x
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ // for hive 3.x
+ new Class<?>[] {
+ Configuration.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ });
+
static IMetaStoreClient createClient(HiveConf hiveConf, String clientClassName) {
+ Method getProxy = null;
+ RuntimeException methodNotFound =
+ new RuntimeException(
+ "Failed to find desired getProxy method from RetryingMetaStoreClient");
+ for (Class<?>[] classes : GET_PROXY_PARAMS) {
+ try {
+ getProxy = RetryingMetaStoreClient.class.getMethod("getProxy", classes);
+ } catch (NoSuchMethodException e) {
+ methodNotFound.addSuppressed(e);
+ }
+ }
+ if (getProxy == null) {
+ throw methodNotFound;
+ }
+
IMetaStoreClient client;
try {
client =
- RetryingMetaStoreClient.getProxy(
- hiveConf,
- tbl -> null,
- new ConcurrentHashMap<>(),
- clientClassName,
- true);
- } catch (MetaException e) {
+ (IMetaStoreClient)
+ getProxy.invoke(
+ null,
+ hiveConf,
Review Comment:
Why does the `Configuration` type parameter `conf` set to null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] tsreaper commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152731838
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -470,17 +472,53 @@ private Lock lock(Identifier identifier) {
return Lock.fromCatalog(lock, identifier);
}
+ private static final List<Class<?>[]> GET_PROXY_PARAMS =
+ Arrays.asList(
+ // for hive 2.x
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ // for hive 3.x
+ new Class<?>[] {
+ Configuration.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ });
+
static IMetaStoreClient createClient(HiveConf hiveConf, String clientClassName) {
+ Method getProxy = null;
+ RuntimeException methodNotFound =
+ new RuntimeException(
+ "Failed to find desired getProxy method from RetryingMetaStoreClient");
+ for (Class<?>[] classes : GET_PROXY_PARAMS) {
+ try {
+ getProxy = RetryingMetaStoreClient.class.getMethod("getProxy", classes);
+ } catch (NoSuchMethodException e) {
+ methodNotFound.addSuppressed(e);
+ }
+ }
+ if (getProxy == null) {
+ throw methodNotFound;
+ }
+
IMetaStoreClient client;
try {
client =
- RetryingMetaStoreClient.getProxy(
- hiveConf,
- tbl -> null,
- new ConcurrentHashMap<>(),
- clientClassName,
- true);
- } catch (MetaException e) {
+ (IMetaStoreClient)
+ getProxy.invoke(
+ null,
+ hiveConf,
Review Comment:
The first parameter of `Method#invoke` is on which object you're calling this method. As this is a static method, this parameter should be `null`. Arguments for the original method starts from the second parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] SteNicholas commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152687901
##########
paimon-hive/paimon-hive-connector-3.1/pom.xml:
##########
@@ -35,15 +35,455 @@ under the License.
<properties>
<hive.version>3.1.2</hive.version>
+ <jackson.version>2.13.3</jackson.version>
</properties>
<dependencies>
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-hive-connector-common</artifactId>
<version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-hive-connector-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-shade</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-flink-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-runtime</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-hive_${scala.binary.version}</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-hive_${scala.binary.version}</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-orc</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-test-utils</artifactId>
+ <version>${test.flink.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.junit.vintage</groupId>
+ <artifactId>junit-vintage-engine</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.module</groupId>
+ <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
+ <version>${jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>com.klarna</groupId>
+ <artifactId>hiverunner</artifactId>
+ <version>${hiverunner.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-jdbc</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-service</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-contrib</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-hcatalog-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive.hcatalog</groupId>
+ <artifactId>hive-webhcat-java-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <!-- This dependency is no longer shipped with the JDK since Java 9.-->
+ <groupId>jdk.tools</groupId>
+ <artifactId>jdk.tools</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-auth</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-annotations</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-hdfs</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-server-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-server-web-proxy</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>jms</artifactId>
+ <groupId>javax.jms</groupId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.reflections</groupId>
+ <artifactId>reflections</artifactId>
+ <version>${reflections.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-service</artifactId>
+ <version>${hive.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <!-- This dependency is no longer shipped with the JDK since Java 9.-->
+ <groupId>jdk.tools</groupId>
+ <artifactId>jdk.tools</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-auth</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-client</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-annotations</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-hdfs</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-registry</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-server-applicationhistoryservice</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-server-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hbase-hadoop-compat</artifactId>
+ <groupId>org.apache.hbase</groupId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.jamon</groupId>
+ <artifactId>jamon-runtime</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive.hcatalog</groupId>
+ <artifactId>hive-hcatalog-core</artifactId>
+ <version>${hive.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-archives</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-annotations</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-hdfs</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>apache-log4j-extras</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.jamon</groupId>
+ <artifactId>jamon-runtime</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hive.hcatalog</groupId>
+ <artifactId>hive-webhcat-java-client</artifactId>
+ <version>${hive.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.jamon</groupId>
+ <artifactId>jamon-runtime</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>jdk.tools</groupId>
+ <artifactId>jdk.tools</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>jms</artifactId>
+ <groupId>javax.jms</groupId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.pentaho</groupId>
+ <artifactId>pentaho-aggdesigner-algorithm</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!--
+ Why we need this test dependency:
+ IDEA reads classes from the same project from target/classes of that module,
+ so even though we've packaged and shaded avro classes into paimon-format.jar
+ we still have to include this test dependency here.
+
+ Why do we put this test dependency before the provided hive-exec:
+ hive-exec produces a shaded jar which contains old versioned avro classes,
+ so we need to make sure that our newer avro is loaded first.
+ -->
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ <version>${avro.version}</version>
+ <scope>test</scope>
</dependency>
+ <!--
+ hive-exec must stay after flink-connector-hive and avro to avoid conflicts
Review Comment:
Does the 3.1 version of `hive-exec` dependency exclude the calcite dependencies? As tested before, the version of calcite dependencies has conflict with flink-table-planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] SteNicholas commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152682434
##########
docs/content/how-to/creating-catalogs.md:
##########
@@ -76,45 +76,11 @@ USE paimon.default;
By using Paimon Hive catalog, changes to the catalog will directly affect the corresponding Hive metastore. Tables created in such catalog can also be accessed directly from Hive.
-### Preparing Paimon Hive Catalog Jar File
-
-Download the jar file with corresponding version.
-
-{{< stable >}}
-
-| Version | Jar |
-|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 2 & 3 | [paimon-hive-catalog-{{< version >}}.jar](https://www.apache.org/dyn/closer.lua/flink/paimon-{{< version >}}/paimon-hive-catalog-{{< version >}}.jar) |
-
-{{< /stable >}}
-
-{{< unstable >}}
-
-| Version | Jar |
-|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 2 & 3 | [paimon-hive-catalog-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-catalog/{{< version >}}/) |
-
-{{< /unstable >}}
-
-You can also manually build bundled jar from the source code.
-
-To build from source code, [clone the git repository]({{< github_repo >}}).
-
-Build bundled jar with the following command.
-`mvn clean install -Dmaven.test.skip=true`
-
-You can find Hive catalog jar in `./paimon-hive/paimon-hive-catalog/target/paimon-hive-catalog-{{< version >}}.jar`.
-
-### Registering Hive Catalog
-
{{< tabs "hive-metastore-example" >}}
{{< tab "Flink" >}}
-To enable Paimon Hive catalog support in Flink, you can pick one of the following two methods.
-
-* Copy Paimon Hive catalog jar file into the `lib` directory of your Flink installation directory. Note that this must be done before starting your Flink cluster.
-* If you're using Flink's SQL client, append `--jar /path/to/paimon-hive-catalog-{{< version >}}.jar` to the starting command of SQL client.
+Paimon Hive catalog in Flink relies on Flink Hive connector bundled jar. You should first download Flink Hive connector bundled jar and add it to classpath. See [here](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/hive/overview/#using-bundled-hive-jar) for more info.
Review Comment:
Is it better to use the master branch link https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/hive/overview/#using-bundled-hive-jar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] SteNicholas commented on a diff in pull request #764: [hive] Remove bundled Hive classes from Paimon Hive catalog and bundle Hive catalog to distributed jars of all engines
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #764:
URL: https://github.com/apache/incubator-paimon/pull/764#discussion_r1152683206
##########
paimon-e2e-tests/pom.xml:
##########
@@ -34,6 +34,7 @@ under the License.
<properties>
<flink.shaded.hadoop.version>2.8.3-10.0</flink.shaded.hadoop.version>
<flink.sql.connector.kafka>flink-sql-connector-kafka</flink.sql.connector.kafka>
+ <flink.sql.connector.hive>flink-sql-connector-hive-2.3.9_${scala.binary.version}</flink.sql.connector.hive>
Review Comment:
Does the e2e tests need to add the test case for 3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org