You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by GitBox <gi...@apache.org> on 2022/11/01 01:58:33 UTC
[GitHub] [incubator-streampark] lvshaokang opened a new pull request, #1938: [Feature] Support Flink 1.16 submit
lvshaokang opened a new pull request, #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938
<!--
Thank you for contributing to StreamPark! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
## Contribution Checklist
- If this is your first time, please read our contributor guidelines: [Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/streampark/issues).
- Name the pull request in the form "[Feature] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- If the PR is unfinished, add `[WIP]` in your PR title, e.g., `[WIP][Feature] Title of the pull request`.
-->
## What changes were proposed in this pull request
Issue Number: close #1620 <!-- REMOVE this line if no issue to close -->
<!--(For example: This pull request proposed to add checkstyle plugin).-->
Support used flink 1.16 to submit job.
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
Add 1.16 shims
## Verifying this change
<!--*(Please pick either of the following options)*-->
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added integration tests for end-to-end.*
- *Added *Test to verify the change.*
- *Manually verified the change by testing locally.* -->
- Manually verified submit job using yarn with custom code/sql by testing locally.
- Verifyed using jar syntax checking.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
wolfboys commented on PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938#issuecomment-1298471058
hi lvshaokang:
thanks for your contribution. I see the shims code for flink-1.16 is exactly the same as flink-1.15, So I don't think it's necessary for us to implement flink-1.16, If the flink version of the user's job is flink-1.16, we can select shims-1.15 for the corresponding shims version in the program, This is just my assumption, you can actually verify whether it is feasible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] lvshaokang commented on a diff in pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
lvshaokang commented on code in PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938#discussion_r1011323795
##########
streampark-flink/streampark-flink-proxy/src/main/scala/org/apache/streampark/flink/proxy/FlinkShimsProxy.scala:
##########
@@ -36,7 +36,7 @@ object FlinkShimsProxy extends Logger {
)
private[this] val SHIMS_PATTERN = Pattern.compile(
- "streampark-flink-shims_flink-(1.12|1.13|1.14|1.15)_(2.11|2.12)-(.*).jar",
+ "streampark-flink-shims_flink-(1.12|1.13|1.14|1.15|1.16)_(2.11|2.12)-(.*).jar",
Review Comment:
> Too many pattern, please extract a const field.
Hi, @1996fanrui . What you mean is that extract regex pattern to a const field?
such this `streampark-flink-shims_flink-(1.12|1.13|1.14|1.15|1.16)_(2.11|2.12)-(.*).jar, (^|.*)streampark-flink-shims_flink-(1.12|1.13|1.14|1.15|1.16)-(.*).jar$`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] lvshaokang commented on pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
lvshaokang commented on PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938#issuecomment-1297914743
cc @wolfboys @1996fanrui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] lvshaokang commented on pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
lvshaokang commented on PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938#issuecomment-1299699829
> So I don't think it's necessary for us to implement flink-1.16, If the flink version of the user's job is flink-1.16, we can select shims-1.15 for the corresponding shims version in the program
@wolfboys , In my test step, i had some problems.
1. I remove all same codes and use the shims 1.15 dependency, but the mvn package jar without any shims 1.15 class
2. So, I modified the maven shade plugin to this,
```
<artifactSet>
<includes>
<include>org.apache.flink:flink-table-api-scala-bridge_${scala.binary.version}</include>
<include>org.apache.streampark:streampark-flink-shims_flink-1.15_${scala.binary.version}</include>
</includes>
</artifactSet>
```
I found the jar file contains the class, but i submit job, it throws a exception following this:
it used the StreamTableEnvironment of 1.15, rather then 1.16
```
Caused by: java.lang.NoSuchMethodError: org.apache.flink.table.catalog.FunctionCatalog.<init>(Lorg/apache/flink/configuration/ReadableConfig;Lorg/apache/flink/table/catalog/CatalogManager;Lorg/apache/flink/table/module/ModuleManager;)V
at org.apache.flink.table.api.bridge.scala.internal.StreamTableEnvironmentImpl$.create(StreamTableEnvironmentImpl.scala:324) ~[streampark-flink-shims_flink-1.16_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.flink.table.api.bridge.scala.StreamTableEnvironment$.create(StreamTableEnvironment.scala:899) ~[streampark-flink-shims_flink-1.16_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.core.FlinkTableInitializer.initEnvironment(FlinkTableInitializer.scala:247) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.core.FlinkTableInitializer$.initialize(FlinkTableInitializer.scala:78) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.core.scala.FlinkStreamTable.init(FlinkStreamTable.scala:54) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.core.scala.FlinkStreamTable.main(FlinkStreamTable.scala:45) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.core.scala.FlinkStreamTable.main$(FlinkStreamTable.scala:44) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.cli.SqlClient$StreamSqlApp$.main(SqlClient.scala:63) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.cli.SqlClient$.delayedEndpoint$org$apache$streampark$flink$cli$SqlClient$1(SqlClient.scala:55) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at org.apache.streampark.flink.cli.SqlClient$delayedInit$body.apply(SqlClient.scala:30) ~[streampark-flink-sqlclient_2.12-1.2.4-SNAPSHOT.jar:1.2.4-SNAPSHOT]
at scala.Function0.apply$mcV$sp(Function0.scala:34) ~[flink-scala_2.12-1.16.0.jar:1.16.0]
at scala.Function0.apply$mcV$sp$(Function0.scala:34) ~[flink-scala_2.12-1.16.0.jar:1.16.0]
```
Meanwhile,I found that the jar package size is a bit larger compared to the previous implementation.
before: 57927
after: 60217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
1996fanrui commented on code in PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938#discussion_r1010381385
##########
streampark-flink/streampark-flink-proxy/src/main/scala/org/apache/streampark/flink/proxy/FlinkShimsProxy.scala:
##########
@@ -36,7 +36,7 @@ object FlinkShimsProxy extends Logger {
)
private[this] val SHIMS_PATTERN = Pattern.compile(
- "streampark-flink-shims_flink-(1.12|1.13|1.14|1.15)_(2.11|2.12)-(.*).jar",
+ "streampark-flink-shims_flink-(1.12|1.13|1.14|1.15|1.16)_(2.11|2.12)-(.*).jar",
Review Comment:
Hi @lvshaokang , thanks for your contribution. Most look good.
Too many pattern, please extract a const field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys merged pull request #1938: [Feature] Support Flink 1.16 submit
Posted by GitBox <gi...@apache.org>.
wolfboys merged PR #1938:
URL: https://github.com/apache/incubator-streampark/pull/1938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org