You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/05 17:49:34 UTC
[GitHub] [flink] MartijnVisser opened a new pull request, #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
MartijnVisser opened a new pull request, #19364:
URL: https://github.com/apache/flink/pull/19364
## What is the purpose of the change
* Upgrade the Hive 2.3 connector to the latest available version, 2.3.9
## Brief change log
* Renamed connector from Hive 2.3.6 to 2.3.9
* Updated dependency
* Updated documentation and build profiles
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1092510526
@luoyuxia Your help would be much appreciated :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on code in PR #19364:
URL: https://github.com/apache/flink/pull/19364#discussion_r845865855
##########
flink-connectors/flink-connector-hive/pom.xml:
##########
@@ -1027,15 +1027,15 @@ under the License.
<profiles>
<!-- Activate these profiles with -Phive-x.x.x to build and test against different Hive versions -->
<profile>
- <id>hive-2.3.6</id>
+ <id>hive-2.3.9</id>
<properties>
- <hive.version>2.3.6</hive.version>
+ <hive.version>2.3.9</hive.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
- <version>${hive-2.3.6-orc-version}</version>
+ <version>${hive-2.3.9-orc-version}</version>
Review Comment:
I suggest to still keep it for test purpose .
It enables the developer to test againset the corresponding Hive version to make sure it works for the specific version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1092597973
@luoyuxia
> I suggest to still keep it for test purpose .
It enables the developer to test againset the corresponding Hive version to make sure it works for the specific version.
I think that's fine, but what should be the value then? Because neither `hive-2.3.6-orc-version` or `hive-2.3.9-orc-version` are defined anywhere in Flink's project. The only definition which seems to be relevant is `<hive-2.2.0-orc-version>1.4.3</hive-2.2.0-orc-version>`. Should we change `hive-2.2.0-orc-version` to `hive-2.3.9-orc-version` and keep the version set at `1.4.3` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser merged PR #19364:
URL: https://github.com/apache/flink/pull/19364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1089117617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f21e0c16f92211144877351735df2f21c0624628",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f21e0c16f92211144877351735df2f21c0624628",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e0c16f92211144877351735df2f21c0624628 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1092595463
> Should we also change `flink-sql-connector-hive-2.3.6` to `flink-sql-connector-hive-3.1.2` in modules-defining-excess-dependencies.modulelist?
Good catch, I'll update `flink-sql-connector-hive-2.3.6` to `flink-sql-connector-hive-2.3.9` there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1090142562
> I am definitely missing some knowledge in Flink's Hive world because I have seen this ticket https://issues.apache.org/jira/browse/FLINK-26739 where I would have thought the version was bumped.
That ticket/PR did add the necessary shims to load/support the additional versions but it should have also included a bump of the Hive connector to the latest version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #19364:
URL: https://github.com/apache/flink/pull/19364#discussion_r843810674
##########
flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml:
##########
@@ -30,8 +30,8 @@ under the License.
<relativePath>..</relativePath>
</parent>
- <artifactId>flink-sql-connector-hive-2.3.6_${scala.binary.version}</artifactId>
- <name>Flink : Connectors : SQL : Hive 2.3.6</name>
+ <artifactId>flink-sql-connector-hive-2.3.9_${scala.binary.version}</artifactId>
Review Comment:
Yes, only for 1.16 since we've also only dropped the old Hive connectors in 1.16 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a diff in pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
fapaul commented on code in PR #19364:
URL: https://github.com/apache/flink/pull/19364#discussion_r843608467
##########
flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml:
##########
@@ -30,8 +30,8 @@ under the License.
<relativePath>..</relativePath>
</parent>
- <artifactId>flink-sql-connector-hive-2.3.6_${scala.binary.version}</artifactId>
- <name>Flink : Connectors : SQL : Hive 2.3.6</name>
+ <artifactId>flink-sql-connector-hive-2.3.9_${scala.binary.version}</artifactId>
Review Comment:
Renaming the module makes backporting difficult. I guess we only do it for 1.16?
##########
flink-connectors/flink-connector-hive/pom.xml:
##########
@@ -1027,15 +1027,15 @@ under the License.
<profiles>
<!-- Activate these profiles with -Phive-x.x.x to build and test against different Hive versions -->
<profile>
- <id>hive-2.3.6</id>
+ <id>hive-2.3.9</id>
<properties>
- <hive.version>2.3.6</hive.version>
+ <hive.version>2.3.9</hive.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
- <version>${hive-2.3.6-orc-version}</version>
+ <version>${hive-2.3.9-orc-version}</version>
Review Comment:
This complete profile seems to be broken. I neither see `${hive-2.3.6-orc-version}` defined somewhere nor `${hive-2.3.9-orc-version}`. Do I miss something here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1092605478
@MartijnVisser Yes, we just change hive-2.2.0-orc-version to hive-2.3.9-orc-version and keep the version set at 1.4.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
wuchong commented on PR #19364:
URL: https://github.com/apache/flink/pull/19364#issuecomment-1091391865
cc @luoyuxia , could you help to review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on a diff in pull request #19364: [FLINK-27063][Connectors/Hive] Upgrade Hive 2.3 connector from 2.3.6 to 2.3.9
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #19364:
URL: https://github.com/apache/flink/pull/19364#discussion_r843818348
##########
flink-connectors/flink-connector-hive/pom.xml:
##########
@@ -1027,15 +1027,15 @@ under the License.
<profiles>
<!-- Activate these profiles with -Phive-x.x.x to build and test against different Hive versions -->
<profile>
- <id>hive-2.3.6</id>
+ <id>hive-2.3.9</id>
<properties>
- <hive.version>2.3.6</hive.version>
+ <hive.version>2.3.9</hive.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
- <version>${hive-2.3.6-orc-version}</version>
+ <version>${hive-2.3.9-orc-version}</version>
Review Comment:
Good one, I don't see it either. Looking back through the Git commits, I'm suspecting that https://github.com/apache/flink/pull/13078/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R150 originally introduced this value as `hive-2.2.0-orc-version`. Probably during a search and replace that one got moved around.
My guess is that we can completely remove this profile and the `hive-2.2.0-orc-version` from the main POM. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org