You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/09 17:18:36 UTC
[GitHub] [incubator-seatunnel] s7monk opened a new pull request, #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
s7monk opened a new pull request, #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046
@https://github.com/apache/incubator-seatunnel/issues/3044
…ovided
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
Modify the scope of flink-shaded-hadoop-2 of the Connector-V2-Iceberg module to provided to be compatible with hadoop3.x
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r990878278
##########
docs/en/connector-v2/source/Rds.md:
##########
@@ -0,0 +1,7 @@
+# Rds
Review Comment:
move to new pull request？
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -120,6 +120,7 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>provided</scope>
Review Comment:
update docs
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Iceberg.md
##########
docs/en/connector-v2/sink/Rds.md:
##########
@@ -0,0 +1,7 @@
+# Rds
Review Comment:
move to new pull request？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1277657473
this pr is ok , please review it@CalvinKirs @EricJoy2048 @hailin0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r993308464
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iceberg-flink-e2e/pom.xml:
##########
@@ -40,6 +40,17 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1001322667
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -120,6 +121,7 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>provided</scope>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1275697950
I have tested the hadoop catalog on hadoop2.7.1 and hadoop3.1.3, and also tested the hive catalog on hive3.1.2, and it runs without problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r993310682
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/pom.xml:
##########
@@ -36,6 +36,7 @@
<module>connector-fake-flink-e2e</module>
<module>connector-mongodb-flink-e2e</module>
<module>connector-iceberg-flink-e2e</module>
+ <module>connector-iceberg-hadoop3-flink-e2e</module>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer merged pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
TyrantLucifer merged PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1272710823
Please add e2e test case in hadoop3.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r993288456
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/pom.xml:
##########
@@ -36,6 +36,7 @@
<module>connector-fake-flink-e2e</module>
<module>connector-mongodb-flink-e2e</module>
<module>connector-iceberg-flink-e2e</module>
+ <module>connector-iceberg-hadoop3-flink-e2e</module>
Review Comment:
Add module: connector-iceberg-hadoop3-spark-e2e
Or
move module to https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r990879478
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -120,6 +120,7 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>provided</scope>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1001322792
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -94,6 +94,7 @@
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<classifier>core</classifier>
+ <scope>provided</scope>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1022252539
##########
docs/en/connector-v2/source/Iceberg.md:
##########
@@ -161,6 +161,19 @@ source {
}
```
+:::tip
+
+In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec and flink-shaded-hadoop-2 in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages.
+
+:::
+
+```
+flink-shaded-hadoop-x-xxx.jar
+hive-exec-xxx.jar
+libfb303-xxx.jar
+```
+Some versions of the hive-exec package do not have libfb303-xxx.jar, so you also need to manually import the Jar package.
+
## Changelog
### 2.2.0-beta 2022-09-26
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1279694151
PTAL @CalvinKirs @hailin0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1000577587
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -94,6 +94,7 @@
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<classifier>core</classifier>
+ <scope>provided</scope>
Review Comment:
Add a note about this into iceberg connector docs?
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -120,6 +121,7 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>provided</scope>
Review Comment:
Add a note about this into iceberg connector docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1026228532
##########
docs/en/connector-v2/source/Iceberg.md:
##########
@@ -161,6 +161,19 @@ source {
}
```
+:::tip
+
+In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec and flink-shaded-hadoop-2 in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages.
+
+:::
+
+```
+flink-shaded-hadoop-x-xxx.jar
+hive-exec-xxx.jar
+libfb303-xxx.jar
+```
+Some versions of the hive-exec package do not have libfb303-xxx.jar, so you also need to manually import the Jar package.
+
## Changelog
### 2.2.0-beta 2022-09-26
Review Comment:
> ok
You can reference other connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r993287162
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iceberg-flink-e2e/pom.xml:
##########
@@ -40,6 +40,17 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
What is the reason for excluding here? If there will be conflicts, it is better that we provide the corresponding shade jar, otherwise this may hide the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1277640951
If this pr is ok, please @ us to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r990878410
##########
seatunnel-connectors-v2/connector-iceberg/pom.xml:
##########
@@ -120,6 +120,7 @@
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>provided</scope>
Review Comment:
update docs
test spark
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Iceberg.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1275691852
I have tested spark with no problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r993292879
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iceberg-flink-e2e/pom.xml:
##########
@@ -40,6 +40,17 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-hadoop-2</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
cc @hailin0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1009216245
##########
docs/en/connector-v2/source/Iceberg.md:
##########
@@ -161,6 +161,19 @@ source {
}
```
+:::tip
+
+In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec and flink-shaded-hadoop-2 in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages.
+
+:::
+
+```
+flink-shaded-hadoop-x-xxx.jar
+hive-exec-xxx.jar
+libfb303-xxx.jar
+```
+Some versions of the hive-exec package do not have libfb303-xxx.jar, so you also need to manually import the Jar package.
+
## Changelog
### 2.2.0-beta 2022-09-26
Review Comment:
Please add the `changed logs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on a diff in pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on code in PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#discussion_r1032014139
##########
docs/en/connector-v2/source/Iceberg.md:
##########
@@ -161,6 +161,19 @@ source {
}
```
+:::tip
+
+In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec and flink-shaded-hadoop-2 in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages.
+
+:::
+
+```
+flink-shaded-hadoop-x-xxx.jar
+hive-exec-xxx.jar
+libfb303-xxx.jar
+```
+Some versions of the hive-exec package do not have libfb303-xxx.jar, so you also need to manually import the Jar package.
+
## Changelog
### 2.2.0-beta 2022-09-26
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] s7monk commented on pull request #3046: [Feature][Connector-V2][Iceberg]Modify the scope of flink-shaded-hadoop-2 to provided to be compatible with hadoop3.x
Posted by GitBox <gi...@apache.org>.
s7monk commented on PR #3046:
URL: https://github.com/apache/incubator-seatunnel/pull/3046#issuecomment-1328030374
PTAL @EricJoy2048 @hailin0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org