You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/05/23 03:57:35 UTC
[GitHub] [incubator-kyuubi] wolfboys opened a new pull request, #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
wolfboys opened a new pull request, #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724
### _Why are the changes needed?
Flink 1.15.0 is out, Kyuubi needs to support it.
Please clarify why the changes are needed. For instance,
1. Upgrade Flink version to 1.15
2. Some API upgrades adapt to Flink 1.15
This part of the work is still in progress, At present, 28 test cases have passed and 9 have failed, You can review, put forward suggestions for improvement, I will continue to improve
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879164841
##########
pom.xml:
##########
@@ -1281,83 +1281,63 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_${scala.binary.version}</artifactId>
+ <artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java</artifactId>
+ <artifactId>flink-sql-client</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
- <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-parser</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-runtime</artifactId>
<version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-client_${scala.binary.version}</artifactId>
+ <artifactId>flink-sql-parser</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
- <exclusions>
Review Comment:
> @wolfboys Could you clarify why remove these exclusions? Thanks!
flink-test-utils just a declare in parent module. When it is really used in `kyuubi-flink-sql-engine` module , dependency's scope is test, and other dependencies will not be attached
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879151579
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
> @wolfboys IIUC, `loop` is a local variable that's only used in one thread. Could you please clarify why it needs to be an `AtomicBoolen`? Thanks!
This is only from the perspective of the characteristics of Scala language and Functional Programming, Variables are declared. It is best not to change, and this principle is followed. my point may be is not true, this is my personal programming habit. avoid using var in scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879223231
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
I agree with @link3280, it's overkill to use AtomicBoolean here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys closed pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys closed pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
URL: https://github.com/apache/incubator-kyuubi/pull/2724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] link3280 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
link3280 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879071835
##########
externals/kyuubi-flink-sql-engine/pom.xml:
##########
@@ -61,19 +61,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
Review Comment:
That's indeed a critical issue. Kyuubi uses some non-public Flink APIs, which have no backward compatibility guarantee. We may need a multi-version solution like FlinkShims in Zeeplin.
@wolfboys Does StreamX use Flink internal APIs as well? How does it achieve multi-version support? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879226845
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
> I agree with @link3280, it's overkill to use AtomicBoolean here
😄 I'll revise it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yanghua commented on pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
yanghua commented on PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#issuecomment-1134642535
@wolfboys please rebase this PR, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] link3280 commented on pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
link3280 commented on PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#issuecomment-1135327603
@wolfboys Thanks for your work! Looking forward to your upcoming PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879012462
##########
pom.xml:
##########
@@ -109,7 +109,7 @@
<curator.version>2.12.0</curator.version>
<delta.version>1.2.1</delta.version>
<fb303.version>0.9.3</fb303.version>
- <flink.version>1.14.4</flink.version>
+ <flink.version>1.15.0</flink.version>
Review Comment:
thx for you review. this time only flink 1.15 is supported. If flink 1.15 and multiple versions are supported at the same time, the development and review time of pr will be extended. For the time being, it can be split into two sub-tasks. This time, flink 1.15 will be supported first, and we will wait for this pr After passing, you can open a file about flink multi-version support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] link3280 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
link3280 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879201139
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
@pan3793 WDYT? Is that recommended in Kyuubi's code style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#issuecomment-1134861648
I'm a little bit surprised that you create dedicated modules for every Flink minor version. Actually, in Kyuubi project, we try our best to avoid maintaining multi-version support in this way, we only adopt this approach in Kyuubi Spark Extension modules because all APIs that we used are private APIs.
IMO, to support both Flink 1.14 & 1.15, even if we access some Flink private API in Flink engine module, it should be handled by reflection properly. In detail, my idea is,
1) Introduce `flink-1.14` and `flink-1.15` maven profiles to handle name changes of some modules.
```
<profile>
<id>flink-1.14</id>
<properties>
<flink.module.scala.suffix></flink.module.scala.suffix>
</properties>
</profile>
<profile>
<id>flink-1.15</id>
<properties>
<flink.module.scala.suffix>${scala.binary.version}</flink.module.scala.suffix>
</properties>
</profile>
```
2) The code should work with `flink-1.14` but may not `flink-1.15`, find them and extract a `FlinkShim` trait with implementations `FlinkShim_1_14` and `FlinkShim_1_15`, to detect and handle the Flink version in runtime.
3) Enable `flink-1.14` and `flink-1.15` in CI
4) Change the default Flink version to 1.15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] link3280 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
link3280 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879076282
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
@wolfboys IIUC, `loop` is a local variable that's only used in one thread. Could you please clarify why it needs to be an `AtomicBoolen`? Thanks!
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -349,6 +347,8 @@ class FlinkOperationSuite extends WithFlinkSQLEngine with HiveJDBCTestHelper {
withJdbcStatement() { statement =>
val metaData = statement.getConnection.getMetaData
var resultSet = metaData.getSchemas(null, null)
+ val DEFAULT_BUILTIN_CATALOG = "default_catalog"
Review Comment:
How about `TableConfigOptions.TABLE_CATALOG_NAME.defaultValue()` instead?
##########
pom.xml:
##########
@@ -1281,83 +1281,63 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_${scala.binary.version}</artifactId>
+ <artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java</artifactId>
+ <artifactId>flink-sql-client</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
- <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-parser</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-runtime</artifactId>
<version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-client_${scala.binary.version}</artifactId>
+ <artifactId>flink-sql-parser</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
- <exclusions>
Review Comment:
@wolfboys Could you clarify why remove these exclusions? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879009578
##########
externals/kyuubi-flink-sql-engine/pom.xml:
##########
@@ -61,19 +61,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
Review Comment:
Based on the discussion, Kyuubi should support 2 or 3 Flink versions, this change will break support for Flink 1.14
##########
externals/kyuubi-flink-sql-engine/pom.xml:
##########
@@ -61,19 +61,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
Review Comment:
Based on the [discussion](https://github.com/apache/incubator-kyuubi/issues/2238#issuecomment-1080370851), Kyuubi should support 2 or 3 Flink versions, this change will break support for Flink 1.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] link3280 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
link3280 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879201139
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/ExecuteStatement.scala:
##########
@@ -129,11 +137,12 @@ class ExecuteStatement(
val resultDescriptor = executor.executeQuery(sessionId, operation)
resultId = resultDescriptor.getResultId
+ val dataTypes = resultDescriptor.getResultSchema.getColumnDataTypes
val rows = new ArrayBuffer[Row]()
- var loop = true
+ val loop = new AtomicBoolean(true)
Review Comment:
@pan3793 WDYT? It's that recommended in Kyuubi's code style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#issuecomment-1134908236
>
In fact, my approach is to refer to the implementation of shims in zeppelin and streamx. If a module is based on a specific flink version such as (flink 1.14) to support multiple different flink versions, use reflection In fact, it will be more troublesome to implement. In addition to the incompatibility of API changes, some versions such as (flink 1.15) will even use new APIs (new APIs than flink 1.14), It's not easy to do by reflection
The current method is only an initial version, and in the final implementation, the common classes and codes in each version will be extracted upward, and only the code compatible with that version will be placed in each shims package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879160125
##########
pom.xml:
##########
@@ -1281,83 +1281,63 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_${scala.binary.version}</artifactId>
+ <artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java</artifactId>
+ <artifactId>flink-sql-client</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
- <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-parser</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-runtime</artifactId>
<version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-sql-client_${scala.binary.version}</artifactId>
+ <artifactId>flink-sql-parser</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
+ <artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
- <exclusions>
Review Comment:
> @wolfboys Could you clarify why remove these exclusions? Thanks!
`flink-test-utils` just a statement here. When it is really used in `kyuubi-flink-sql-engine` , dependency's scope is test, and other dependencies will not be attached
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -349,6 +347,8 @@ class FlinkOperationSuite extends WithFlinkSQLEngine with HiveJDBCTestHelper {
withJdbcStatement() { statement =>
val metaData = statement.getConnection.getMetaData
var resultSet = metaData.getSchemas(null, null)
+ val DEFAULT_BUILTIN_CATALOG = "default_catalog"
Review Comment:
good idea. I agree with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879131106
##########
externals/kyuubi-flink-sql-engine/pom.xml:
##########
@@ -61,19 +61,19 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
Review Comment:
> That's indeed a critical issue. Kyuubi uses some non-public Flink APIs, which have no backward compatibility guarantee. We may need a multi-version solution like FlinkShims in Zeeplin.
>
> @wolfboys Does StreamX use Flink internal APIs as well? How does it achieve multi-version support? Thanks!
Since I am not familiar with the Kyuubi project, I originally wanted to upgrade the version to Flink 1.15, implement this, and then consider multi version support. Now it seems that upgrading directly to 1.15 will affect the support of the released Flink 1.14. I will bring up the PR supported by multiple versions later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#issuecomment-1134913565
> I'm a little bit surprised that you create dedicated modules for every Flink minor version. Actually, in Kyuubi project, we try our best to avoid maintaining multi-version support in this way, we only adopt this approach in Kyuubi Spark Extension modules because all APIs that we used are private.
>
> IMO, to support both Flink 1.14 & 1.15, even if we access some Flink private API in Flink engine module, it should be handled by reflection properly. In detail, my idea is,
>
> 1. Introduce `flink-1.14` and `flink-1.15` maven profiles to handle version/name changes of some modules.
>
> ```
> <profile>
> <id>flink-1.14</id>
> <properties>
> <flink.version>1.14.4</flink.version>
> <flink.module.scala.suffix>_${scala.binary.version}</flink.module.scala.suffix>
> </properties>
> </profile>
> <profile>
> <id>flink-1.15</id>
> <properties>
> <flink.version>1.15.0</flink.version>
> <flink.module.scala.suffix></flink.module.scala.suffix>
> </properties>
> </profile>
> ```
>
> 2. The code should work with `flink-1.14` but may not `flink-1.15`, find them and extract a `FlinkShim` trait with implementations `FlinkShim_1_14` and `FlinkShim_1_15`, to detect and handle the Flink version in runtime.
> 3. Enable `flink-1.14` and `flink-1.15` in CI
> 4. Change the default Flink version to 1.15
>
> On other hand, if the API changes are huge enough, I agree with you that dedicated modules for each version is a better way.
Thank you for your review and so great point of view, I will seriously consider how to achieve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wolfboys commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
wolfboys commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879012462
##########
pom.xml:
##########
@@ -109,7 +109,7 @@
<curator.version>2.12.0</curator.version>
<delta.version>1.2.1</delta.version>
<fb303.version>0.9.3</fb303.version>
- <flink.version>1.14.4</flink.version>
+ <flink.version>1.15.0</flink.version>
Review Comment:
thx for you review. this time only flink 1.15 is supported. If flink 1.15 and multiple versions are supported at the same time, the development and review time of pr will be extended. For the time being, it can be split into two sub-tasks. This time, flink 1.15 will be supported first, and we will wait for this pr After passing, we can open a issue about flink multi-version support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #2724: [KYUUBI #2238] [WIP] Support Flink 1.15
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #2724:
URL: https://github.com/apache/incubator-kyuubi/pull/2724#discussion_r879010513
##########
pom.xml:
##########
@@ -109,7 +109,7 @@
<curator.version>2.12.0</curator.version>
<delta.version>1.2.1</delta.version>
<fb303.version>0.9.3</fb303.version>
- <flink.version>1.14.4</flink.version>
+ <flink.version>1.15.0</flink.version>
Review Comment:
We need add a `flink-1.15` maven profile to overwrite the `flink.version` property rather than upgrade it in place, and once we implement support everything on the new Flink version, we can change the default property here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org