You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "FangYongs (via GitHub)" <gi...@apache.org> on 2023/03/02 09:09:39 UTC
[GitHub] [flink-table-store] FangYongs opened a new pull request, #569: [FLINK-31269] Split hive connector to each module of each version
FangYongs opened a new pull request, #569:
URL: https://github.com/apache/flink-table-store/pull/569
This pr aims to split hive connector to each module of each version. Besides connector, there are different versions of hive catalog (hive 2 and 3). Hive3 metastore can be connected through the client of hive2.3.9. We do not need to create a independent modules for catalog. If necessary in the future, we can also divide the hive catalog into sub-modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1459580158
> >
>
> Have you tried to setup metastore in hive-2.1-cdh-6.3, and use flink-table-store-hive-catalog.jar in Flink side?
Hi @JingsongLi I also test for hive-2.1.1, it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi closed pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #569: [FLINK-31269] Split hive connector to each module of each version
URL: https://github.com/apache/flink-table-store/pull/569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper merged pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper merged PR #569:
URL: https://github.com/apache/flink-table-store/pull/569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1453120013
I'm mainly worried about whether Hive-catalog can work properly. Have you tested all Hive versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1453209230
>
@JingsongLi I setup `metastore` in `Hive3.1.2` and use `flink-table-store-hive-catalog.jar` which is compiled by `Hive-2.3.9` in flink to execute `DDL` such as `CREATE` and `SHOW TABLES`, it works. Of course, splitting the catalog into different versions is also an option, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#discussion_r1126129142
##########
docs/content/docs/engines/hive.md:
##########
@@ -42,12 +42,12 @@ Table Store currently supports MR and Tez execution engine for Hive.
Download the jar file with corresponding version.
-| |Jar|
-|---|---|
-|Hive 3.1|[flink-table-store-hive-connector-{{< version >}}_3.1.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_3.1.jar)|
-|Hive 2.3|[flink-table-store-hive-connector-{{< version >}}_2.3.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.3.jar)|
-|Hive 2.2|[flink-table-store-hive-connector-{{< version >}}_2.2.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.2.jar)|
-|Hive 2.1|[flink-table-store-hive-connector-{{< version >}}_2.1.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.1.jar)|
+| | Jar |
+|---|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|Hive 3.1| [flink-table-store-hive-connector-3.1-{{< version >}}.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_3.1.jar) |
+|Hive 2.3| [flink-table-store-hive-connector-2.3-{{< version >}}.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.3.jar) |
+|Hive 2.2| [flink-table-store-hive-connector-2.2-{{< version >}}.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.2.jar) |
+|Hive 2.1| [flink-table-store-hive-connector-2.1-{{< version >}}.jar](https://www.apache.org/dyn/closer.lua/flink/flink-table-store-{{< version >}}/flink-table-store-hive-connector-{{< version >}}_2.1.jar) |
Review Comment:
As we now have a separate module for Hive 2.1 CDH 6.3, we should also add a row for that version in the document.
##########
.github/workflows/build-different-versions.yml:
##########
@@ -16,12 +16,3 @@ jobs:
- name: Build
run: |
./mvnw clean install -Dmaven.test.skip=true
- - name: Build Hive 3.1
- run: |
- ./mvnw clean install -Dmaven.test.skip=true -Phive-3.1 -f flink-table-store-hive
- - name: Build Hive 2.2
- run: |
- ./mvnw clean install -Dmaven.test.skip=true -Phive-2.2 -f flink-table-store-hive
- - name: Build Hive 2.1
- run: |
- ./mvnw clean install -Dmaven.test.skip=true -Phive-2.1 -f flink-table-store-hive
Review Comment:
Remove this whole file. No more different versions to check now.
##########
docs/content/docs/engines/hive.md:
##########
@@ -62,16 +62,9 @@ You are using an unreleased version of Table Store so you need to manually build
To build from source code, either [download the source of a release](https://flink.apache.org/downloads.html) or [clone the git repository]({{< github_repo >}}).
Build bundled jar with the following command.
+`mvn clean install -Dmaven.test.skip=true`
-| Version | Command |
-|---|---|
-| Hive 3.1 | `mvn clean install -Dmaven.test.skip=true -Phive-3.1` |
-| Hive 2.3 | `mvn clean install -Dmaven.test.skip=true` |
-| Hive 2.2 | `mvn clean install -Dmaven.test.skip=true -Phive-2.2` |
-| Hive 2.1 | `mvn clean install -Dmaven.test.skip=true -Phive-2.1` |
-| Hive 2.1 CDH 6.3 | `mvn clean install -Dmaven.test.skip=true -Phive-2.1-cdh-6.3` |
-
-You can find Hive connector jar in `./flink-table-store-hive/flink-table-store-hive-connector/target/flink-table-store-hive-connector-{{< version >}}.jar`.
+For hive-2.3, you can find Hive connector jar in `./flink-table-store-hive/flink-table-store-hive-connector-2.3/target/flink-table-store-hive-connector-2.3-{{< version >}}.jar`.
Review Comment:
```suggestion
You can find Hive connector jar in `./flink-table-store-hive/flink-table-store-hive-connector-2.3/target/flink-table-store-hive-connector-<hive-version>-{{< version >}}.jar`.
```
I also suggest changing the description in Flink's document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1465398573
Thanks @tsreaper Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1455307283
>
Have you tried to setup metastore in hive-2.1-cdh-6.3, and use flink-table-store-hive-catalog.jar in Flink side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#issuecomment-1459579297
Thanks @tsreaper DONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #569: [FLINK-31269] Split hive connector to each module of each version
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #569:
URL: https://github.com/apache/flink-table-store/pull/569#discussion_r1129250207
##########
docs/content/docs/engines/hive.md:
##########
@@ -137,7 +131,7 @@ SELECT * FROM test_table;
Run the following Hive SQL in Hive CLI to access the created table.
```sql
--- Assume that flink-table-store-hive-connector-{{< version >}}.jar is already in auxlib directory.
+-- Assume that flink-table-store-hive-connector-2.3-{{< version >}}.jar is already in auxlib directory.
Review Comment:
`2.3` -> `<hive-version>`. There are other appearance of `flink-table-store-hive-connector-{{< version >}}.jar` in the document. Please replace them.
##########
flink-table-store-hive/flink-table-store-hive-connector-2.1-cdh-6.3/pom.xml:
##########
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>flink-table-store-hive</artifactId>
+ <groupId>org.apache.flink</groupId>
+ <version>0.4-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>flink-table-store-hive-connector-2.1-cdh-6.3</artifactId>
+ <name>Flink Table Store : Hive Connector : 2.1-cdh-6.3</name>
+
+ <packaging>jar</packaging>
+
+ <properties>
+ <hive.version>2.1.1-cdh6.3.4</hive.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-store-hive-connector</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>shade-flink</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.flink:flink-table-store-hive-connector</include>
Review Comment:
Rename the old module to `flink-table-store-hive-connector-common`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org