You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "lvshaokang (via GitHub)" <gi...@apache.org> on 2023/03/30 07:55:49 UTC

[GitHub] [incubator-streampark] lvshaokang opened a new pull request, #2533: [Feature] Support Flink 1.17 application submit

lvshaokang opened a new pull request, #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533

   <!--
   Thank you for contributing to StreamPark! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   ## Contribution Checklist
   
     - If this is your first time, please read our contributor guidelines: [Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
   
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-streampark/issues).
   
     - Name the pull request in the form "[Feature] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
   
     - If the PR is unfinished, add `[WIP]` in your PR title, e.g., `[WIP][Feature] Title of the pull request`.
   
   -->
   
   ## What changes were proposed in this pull request
   
   Issue Number: close #2503  <!-- REMOVE this line if no issue to close -->
   
   <!--(For example: This pull request proposed to add checkstyle plugin).-->
   
   ## Brief change log
   
   <!--*(for example:)*
   - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   
   add streampark-flink-shims_flink-1.17 module 
   
   ## Verifying this change
   
   Manually verified the change by testing locally.
   
   <!--*(example:)*
   - *Added integration tests for end-to-end.*
   - *Added *Test to verify the change.*
   - *Manually verified the change by testing locally.* -->
   
   ## Does this pull request potentially affect one of the following parts
    - Dependencies (does it add or upgrade a dependency): (yes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-streampark] wolfboys merged pull request #2533: [Feature] Support Flink 1.17 application submit

Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys merged PR #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-streampark] lvshaokang commented on pull request #2533: [Feature] Support Flink 1.17 application submit

Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533#issuecomment-1489889556

   Manually test snapshot the following as:
   tested list:
   1. yarn application/session submit
   2. k8s application submit
   
   <img width="1471" alt="17_run" src="https://user-images.githubusercontent.com/29084491/228769104-2c788713-5d10-4ce6-9c84-b91ad95d13a0.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2533: [Feature] Support Flink 1.17 application submit

Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533#discussion_r1153908974


##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/impl/YarnApplicationClient.scala:
##########
@@ -74,6 +74,7 @@ object YarnApplicationClient extends YarnClientTrait {
             case Array(1, 14, _) => array += s"${workspace.APP_SHIMS}/flink-1.14"
             case Array(1, 15, _) => array += s"${workspace.APP_SHIMS}/flink-1.15"
             case Array(1, 16, _) => array += s"${workspace.APP_SHIMS}/flink-1.16"
+            case Array(1, 17, _) => array += s"${workspace.APP_SHIMS}/flink-1.17"
             case _ => throw new UnsupportedOperationException(s"Unsupported flink version: ${submitRequest.flinkVersion}")
           }

Review Comment:
   We can improve flink version adaptation code:       
      ```
   version match {
               case Array(1, v, _) if v >= 12 && v <= 17 => array += s"${workspace.APP_SHIMS}/flink-1.$v"
               case _ => throw new UnsupportedOperationException(s"Unsupported flink version: ${submitRequest.flinkVersion}")
             }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2533: [Feature] Support Flink 1.17 application submit

Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533#discussion_r1153918408


##########
streampark-flink/streampark-flink-proxy/src/main/scala/org/apache/streampark/flink/proxy/FlinkShimsProxy.scala:
##########
@@ -34,7 +34,7 @@ object FlinkShimsProxy extends Logger {
     Pattern.CASE_INSENSITIVE | Pattern.DOTALL)
 
   private[this] val SHIMS_PATTERN = Pattern.compile(
-    "streampark-flink-shims_flink-(1.12|1.13|1.14|1.15|1.16)_(2.11|2.12)-(.*).jar",
+    ConfigConst.STREAMPARK_FLINK_SHIMS_JAR_REGEX,

Review Comment:
   I suggest add a new method, so we don’t need to change the code  when support a new flink version
   ```
     private[this] def getShimsLibPattern(flinkVersion: FlinkVersion): Pattern = {
       val array = flinkVersion.version.split("\\.").map(_.trim.toInt)
       val reg = array match {
         case Array(1, v, _) if v >= 12 && v <= 14 =>
           s"streampark-flink-shims_flink-1.${v}_(2.11|2.12)-(.*).jar"
         case Array(1, v, _) if v >= 15 =>
           s"streampark-flink-shims_flink-1.${v}_2.12-(.*).jar"
         case _ => throw new UnsupportedOperationException(s"Unsupported flink version: $flinkVersion")
       }
       Pattern.compile(reg, Pattern.CASE_INSENSITIVE | Pattern.DOTALL)
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2533: [Feature] Support Flink 1.17 application submit

Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2533:
URL: https://github.com/apache/incubator-streampark/pull/2533#discussion_r1153909564


##########
streampark-flink/streampark-flink-packer/src/main/scala/org/apache/streampark/flink/packer/pipeline/BuildRequest.scala:
##########
@@ -64,6 +64,7 @@ sealed trait FlinkBuildParam extends BuildParam {
           case Array(1, 14, _) => s"${localWorkspace.APP_SHIMS}/flink-1.14"
           case Array(1, 15, _) => s"${localWorkspace.APP_SHIMS}/flink-1.15"
           case Array(1, 16, _) => s"${localWorkspace.APP_SHIMS}/flink-1.16"
+          case Array(1, 17, _) => s"${localWorkspace.APP_SHIMS}/flink-1.17"
           case _ => throw new UnsupportedOperationException(s"Unsupported flink version: $flinkVersion")
         }
       }

Review Comment:
   We can improve flink version adaptation code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org