You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampark.apache.org by ta...@apache.org on 2026/08/04 08:23:58 UTC
(streampark) branch dev updated: [Flink] Drop 1.12–1.16 shims, baseline 1.17+, add 2.x support (#4419)
This is an automated email from the ASF dual-hosted git repository.
GuoNingNing pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/streampark.git
The following commit(s) were added to refs/heads/dev by this push:
new e770d2e8e [Flink] Drop 1.12–1.16 shims, baseline 1.17+, add 2.x support (#4419)
e770d2e8e is described below
commit e770d2e8e17fdeacddff62bf306f3235e136dca7
Author: Assert <42...@users.noreply.github.com>
AuthorDate: Tue Aug 4 16:23:51 2026 +0800
[Flink] Drop 1.12–1.16 shims, baseline 1.17+, add 2.x support (#4419)
* [Build] Raise StreamPark compile and CI baseline to JDK 11
Set Maven release 11, update Spotless compliance, switch Docker and CI
workflows to JDK 11, and align module compiler overrides.
Closes #4409 (part 3/3). Ref #4410.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix ClassLoaderUtils dynamic classpath on JDK 9+
Walk the class hierarchy to locate the ucp field on JDK 11+ app
classloaders, add JUnit coverage, and enable surefire add-opens on JDK 9+.
Closes #4409 (part 1/3). Ref #4410.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Console] JDK 11 runtime readiness for StreamPark Console
Enable add-opens JVM opts, enforce JDK 11+ at startup, fix classpath layout,
add javax.annotation-api, and document Console vs job JDK requirements.
Closes #4409 (part 2/3). Ref #4410.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Build][E2E] Fix JDK 11 MavenWrapper and stabilize local basic E2E
Compile MavenWrapperHelper with --release 11 at runtime and stop
packaging prebuilt wrapper classes so project builds work in JDK 11
containers. Update basic E2E for Docker 29, local Chrome, Git SSL,
and Projects page flows verified on macOS.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Remove Scala on JDK 11 baseline for StreamPark 3.0
Migrate all modules from Scala to Java, drop scala-maven-plugin and
_2.12 artifact suffixes, and consolidate the JDK 11 platform baseline
with a zero-Scala codebase for Flink, Spark, Console, and connectors.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Drop 1.12–1.16 shims, baseline 1.17+, add 2.0/2.1/2.2 support
Remove legacy Flink 1.12–1.16 shims, keep 1.17–1.20 as the 1.x baseline,
and add Java shims-base-v2 with Flink 2.0/2.1/2.2 modules. Update
FlinkVersion, shims classloading, console packaging, and UI hints for
the new supported version matrix.
Generated-by: Cursor
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Bump common module version to 3.0.0-preview
Align common and shaded parent POM versions and CLI banner with StreamPark 3.0 preview.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Bump Flink module version to 3.0.0-preview
Update Flink parent and submodule POM versions for StreamPark 3.0 preview.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix Sonar security findings after Scala-to-Java migration
Replace printStackTrace with structured logging and document MD5 usage as
non-cryptographic file checksums to satisfy SonarCloud security rules.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Fix Sonar security findings in Flink modules
Suppress non-cryptographic MD5 checksum usage, remove printStackTrace from
ParameterCli, and document safe dynamic ClickHouse batch SQL assembly.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix Sonar security findings for path handling and debug output
Add canonical path helpers, resolve jar uploads under trusted base dirs,
replace printStackTrace in Flink shims, and suppress non-crypto MD5/SQL checks.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Harden path validation and fix Sonar reliability findings
Use Path-based config resolution, refactor AutoCloseUtils to try-with-resources,
and consolidate ClickHouse SQL suppressions at method level.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Address Sonar path injection review comments
Introduce SafePathUtils with Path.resolve-based sanitization, remove
filesystem oracle checks, and add NOSONAR markers for non-crypto MD5/SQL usage.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix Sonar path I/O taint and reliability findings
Co-locate Path.resolve with file I/O in SafePathUtils, reuse ThreadLocalRandom
for Redis endpoint selection, and exclude versioned shims from CPD checks.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Add NOSONAR for validated config/jar path I/O
Sonar security engine does not propagate Path.resolve sanitization across
methods; explicit NOSONAR marks intentional user-supplied config paths.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix Sonar reliability bugs across common and flink modules
Handle InterruptedException correctly, replace unsafe regex patterns,
fix Optional access, and expand Sonar CPD/issue suppressions for migration code.
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Fix remaining Sonar bugs and add SonarCloud CPD exclusions
Replace DateUtils regex parsing, fix Optional and InterruptedException
handling, and use .sonarcloud.properties for automatic analysis CPD scope.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Throw IllegalArgumentException in FlinkTableInitializerV2 SQL init
Fix Sonar S3984 by propagating YAML SQL initialization failures.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Apply Spotless formatting and fix FlinkCheckpointWatcher syntax
Run spotless:apply across all Java modules to satisfy CI code-style checks.
Fix missing parenthesis in FlinkCheckpointWatcher that blocked Spotless
removeUnusedImports parsing.
Co-authored-by: Cursor <cu...@cursor.com>
* [CI] Sync workflow config with StreamPark 3.0 baseline
- Ignore webapp-v2 paths in Backend and Unit-Test workflows
- Bump dorny/paths-filter action pin
- Fail E2E workflow when test jobs fail (exit 1)
Co-authored-by: Cursor <cu...@cursor.com>
* [CI] Sync workflow config with StreamPark 3.0 baseline
- Ignore webapp-v2 paths in Backend and Unit-Test workflows
- Bump dorny/paths-filter action pin
- Fail E2E workflow when test jobs fail (exit 1)
Co-authored-by: Cursor <cu...@cursor.com>
* [CI] Add ASF license headers and SafePathUtils Sonar ignore rules
Co-authored-by: Cursor <cu...@cursor.com>
* [CI] Add ASF license headers and SafePathUtils Sonar ignore rules
Co-authored-by: Cursor <cu...@cursor.com>
* [Common] Address Sonar issues after Scala-to-Java migration
Refactor ConfigKeys/CommonConfig/Workspace to Java naming conventions,
harden regex patterns, fix compile issues in SafePathUtils and related
utilities, and expand SonarCloud ignore rules for intentional shims patterns.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Align project version to 3.0.0-SNAPSHOT
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Align PR #4419 with post-Scala Java artifact model
Revert dev merge's _2.12 suffix and scala-bridge wiring for Flink modules.
Restore no-suffix StreamPark artifacts (streampark-common, shims, client)
and Java-only 1.17–1.20 shims poms; keep Flink 2.0–2.3 support.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Finish no-scala artifact and FlinkVersion cleanup
Remove JavaConverters from FlinkVersion and align remaining POM/sources
with post-Scala artifact IDs before merging latest dev.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Reduce shims duplication via layered base architecture
Reuse shims-base from shims-base-v2 instead of duplicating identical
sources, move shared 1.x/2.x context and client classes into base
modules, and keep version modules as thin Flink API tier extensions only.
Also harden FlinkSqlValidator against null operands and empty parse results.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Revert aggressive shims dedup; keep layered base reuse
Roll back version-module and trait restructuring done for Sonar CPD.
Retain only natural dedup: shims-base-v2 depends on shims-base for
identical shared sources, plus FlinkSqlValidator null-safe fixes.
Co-authored-by: Cursor <cu...@cursor.com>
* [CI] Exclude Flink shims from SonarCloud duplication analysis
Multi-version shims modules are structurally similar by design; exclude
them from CPD so the quality gate reflects maintainable duplication only.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Fix SonarCloud new code smells for PR #4419
Address deprecated API annotations, null collections, exception types,
and minor refactors across Flink shims, Spark modules, and K8s parsers
to reduce new issues reported by SonarCloud.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Refactor Sonar fixes without SuppressWarnings masking
Replace java:S* suppressions with real refactors: command handler map for
FlinkSqlExecutor, split SqlSplitter/ParameterCli methods, JsonNode-based
FlinkRestModels parsers, and simplified RegExpTest patterns.
Co-authored-by: Cursor <cu...@cursor.com>
* [Spark] Restore streampark-spark to match dev branch
Revert accidental Scala-to-Java migration artifacts so Spark modules stay unchanged on this Flink-focused branch.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Fix build and shims test issues from compile verification
Restore k8s Jackson deps, guard null planner type in table initializer, fix 2.x TableExt compile error, complete shims-test dependencies, and update YarnPerJobTestCase for SubmitResponse API.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Fix Sonar reliability bug in FlinkSqlExecutor.setConfig
Guard SET command operand access to prevent ArrayIndexOutOfBoundsException (javabugs:S6466) and align SqlClient runtime-mode parsing with the same length check.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Fix root POM full build after common artifact rename
Restore scala-bridge module, align Spark/console POM artifact IDs with
post-migration naming, and exclude webapp-v2 from RAT checks.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Remove leftover connector migration script without license header
Drop one-off tools/migrate_connector_phase6.py to fix License header CI.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Restore packer test deps and workspace config after Java migration
Add httpclient test scope dependency and surefire workspace property so
MavenToolTest can resolve artifacts in CI and local builds.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Defer YarnPerJobTestCase init until FLINK_HOME is set
Lazy-initialize the manual YARN harness so CI skips the test instead of
failing during static class initialization when FLINK_HOME is absent.
Co-authored-by: Cursor <cu...@cursor.com>
* [Flink] Replace YarnPerJobTestCase skip with harness compatibility checks
Verify deployInternal reflection, SubmitResponse, yarn-per-job CLI parsing,
and optionally FLINK_HOME layout when the env var is present.
Co-authored-by: Cursor <cu...@cursor.com>
* [Build] Remove stale scala runtime jars from known-dependencies
Dist packaging no longer ships scala-compiler/library/reflect as
third-party artifacts; align known-dependencies with check-LICENSE.sh.
Co-authored-by: Cursor <cu...@cursor.com>
* [Console] Restore scala runtime deps in dist assembly
Console service still loads Scala code at runtime; without these jars the
dist package fails to start (ClassNotFoundException: scala.Product). Re-add
the scala entries to known-dependencies.txt to match the packaged lib.
Co-authored-by: Cursor <cu...@cursor.com>
---------
Co-authored-by: Cursor <cu...@cursor.com>
---
.sonarcloud.properties | 18 +
pom.xml | 5 +-
streampark-common-scala-bridge/pom.xml | 2 +-
streampark-common/pom.xml | 6 +-
.../streampark/common/conf/FlinkVersion.java | 140 ++++--
.../streampark-console-service/pom.xml | 138 ++++--
.../console/core/runner/EnvInitializer.java | 2 +-
.../e2e/cases/ProjectsManagementTest.java | 2 +-
streampark-flink/pom.xml | 9 +
.../streampark-flink-client-api/pom.xml | 30 +-
.../streampark-flink-client-core/pom.xml | 36 +-
.../flink/client/test/YarnPerJobTestCase.java | 249 ++++++++++
.../streampark-flink-kubernetes/pom.xml | 12 +-
.../flink/kubernetes/watcher/FlinkRestModels.java | 290 ++++++++++++
streampark-flink/streampark-flink-packer/pom.xml | 15 +-
.../flink/packer/docker/DockerClients.java | 77 ++++
streampark-flink/streampark-flink-proxy/pom.xml | 10 +-
streampark-flink/streampark-flink-shims/pom.xml | 22 +-
.../pom.xml | 41 +-
.../flink/core/FlinkParameterUtils.java} | 49 +-
.../streampark/flink/core/FlinkSqlExecutor.java | 293 ++++++++++++
.../flink/core/FlinkStreamTableTraitV2.java} | 511 ++++++++-------------
.../flink/core/FlinkStreamingInitializerV2.java | 193 ++++++++
.../flink/core/FlinkTableInitializerV2.java | 297 ++++++++++++
.../streampark/flink/core/FlinkTableTrait.java | 262 ++++++-----
.../apache/streampark/flink/core/SqlCommand.java | 100 ++--
.../flink/core/StreamEnvConfigFunction.java} | 17 +-
.../flink/core/TableEnvConfigFunction.java} | 17 +-
.../flink/core/conf/FlinkConfiguration.java | 42 +-
.../apache/streampark/flink/util/FlinkUtils.java | 28 +-
.../java/org/apache/streampark/RegExpTest.java | 92 ++++
.../streampark-flink-shims-base/pom.xml | 25 +-
.../streampark/flink/core/FlinkClientTrait.java | 22 +-
.../flink/core/FlinkKubernetesClientTrait.java | 1 +
...ironmentUtils.java => FlinkParameterUtils.java} | 43 +-
.../streampark/flink/core/FlinkSqlExecutor.java | 359 ++++++++-------
.../flink/core/FlinkSqlValidationResult.java | 182 ++++----
.../streampark/flink/core/FlinkSqlValidator.java | 229 ++++-----
.../flink/core/FlinkStreamTableTrait.java | 435 +++++++++---------
.../flink/core/FlinkStreamingInitializer.java | 167 +++----
.../flink/core/FlinkTableInitializer.java | 241 +++++-----
.../streampark/flink/core/FlinkTableTrait.java | 262 ++++++-----
.../apache/streampark/flink/core/SqlCommand.java | 108 +++--
.../streampark/flink/core/SqlCommandCall.java | 50 +-
.../flink/core/SqlCommandConverter.java} | 10 +-
.../flink/core/SqlCommandConverters.java | 49 +-
.../streampark/flink/core/SqlCommandParser.java | 100 ++--
.../apache/streampark/flink/core/SqlSegment.java | 32 +-
.../apache/streampark/flink/core/SqlSplitter.java | 425 +++++++++--------
.../streampark/flink/core/StreamEnvConfig.java | 23 +-
.../flink/core/StreamTableEnvConfig.java | 37 +-
.../streampark/flink/core/TableEnvConfig.java | 23 +-
.../flink/core/conf/FlinkConfiguration.java | 40 +-
.../streampark/flink/core/conf/FlinkRunOption.java | 35 +-
.../streampark/flink/core/conf/ParameterCli.java | 213 ++++-----
.../apache/streampark/flink/util/FlinkUtils.java | 26 +-
.../streampark-flink-shims-test/pom.xml | 50 +-
.../streampark/flink/core/StreamTableContext.java | 103 -----
.../apache/streampark/flink/core/TableContext.java | 88 ----
.../org/apache/streampark/flink/core/TableExt.java | 87 ----
.../streampark/flink/core/FlinkClusterClient.java | 27 --
.../flink/core/FlinkKubernetesClient.java | 27 --
.../streampark/flink/core/StreamTableContext.java | 103 -----
.../apache/streampark/flink/core/TableContext.java | 88 ----
.../org/apache/streampark/flink/core/TableExt.java | 87 ----
.../flink/core/FlinkKubernetesClient.java | 27 --
.../streampark/flink/core/StreamTableContext.java | 92 ----
.../apache/streampark/flink/core/TableContext.java | 81 ----
.../org/apache/streampark/flink/core/TableExt.java | 87 ----
.../streampark-flink-shims_flink-1.15/pom.xml | 144 ------
.../streampark/flink/core/StreamTableContext.java | 49 --
.../apache/streampark/flink/core/TableContext.java | 43 --
.../org/apache/streampark/flink/core/TableExt.java | 70 ---
.../streampark-flink-shims_flink-1.16/pom.xml | 154 -------
.../flink/core/FlinkKubernetesClient.java | 37 --
.../streampark/flink/core/StreamTableContext.java | 49 --
.../apache/streampark/flink/core/TableContext.java | 38 --
.../org/apache/streampark/flink/core/TableExt.java | 70 ---
.../streampark-flink-shims_flink-1.17/pom.xml | 54 +--
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 4 +-
.../streampark/flink/core/StreamTableContext.java | 22 +-
.../apache/streampark/flink/core/TableContext.java | 11 +-
.../org/apache/streampark/flink/core/TableExt.java | 52 +--
.../streampark-flink-shims_flink-1.18/pom.xml | 58 +--
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 22 +-
.../apache/streampark/flink/core/TableContext.java | 11 +-
.../org/apache/streampark/flink/core/TableExt.java | 52 +--
.../streampark-flink-shims_flink-1.19/pom.xml | 57 +--
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 22 +-
.../apache/streampark/flink/core/TableContext.java | 11 +-
.../org/apache/streampark/flink/core/TableExt.java | 52 +--
.../streampark-flink-shims_flink-1.20/pom.xml | 57 +--
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 22 +-
.../apache/streampark/flink/core/TableContext.java | 11 +-
.../org/apache/streampark/flink/core/TableExt.java | 51 +-
.../pom.xml | 43 +-
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 4 +-
.../streampark/flink/core/StreamTableContext.java | 291 ++++++++++++
.../apache/streampark/flink/core/TableContext.java | 166 +++++++
.../org/apache/streampark/flink/core/TableExt.java | 62 +++
.../pom.xml | 43 +-
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 349 ++++++++++++++
.../apache/streampark/flink/core/TableContext.java | 224 +++++++++
.../org/apache/streampark/flink/core/TableExt.java | 62 +++
.../pom.xml | 42 +-
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 387 ++++++++++++++++
.../apache/streampark/flink/core/TableContext.java | 262 +++++++++++
.../org/apache/streampark/flink/core/TableExt.java | 62 +++
.../pom.xml | 42 +-
.../streampark/flink/core/FlinkClusterClient.java | 1 +
.../flink/core/FlinkKubernetesClient.java | 1 +
.../streampark/flink/core/StreamTableContext.java | 399 ++++++++++++++++
.../apache/streampark/flink/core/TableContext.java | 262 +++++++++++
.../org/apache/streampark/flink/core/TableExt.java | 62 +++
.../streampark-flink-sqlclient/pom.xml | 26 +-
.../org/apache/streampark/flink/cli/SqlClient.java | 21 +-
streampark-flink/streampark-flink-udf/pom.xml | 8 +-
streampark-spark/pom.xml | 2 +-
.../streampark-spark-client-api/pom.xml | 4 +-
.../streampark-spark-client-core/pom.xml | 2 +-
132 files changed, 6798 insertions(+), 4353 deletions(-)
diff --git a/.sonarcloud.properties b/.sonarcloud.properties
new file mode 100644
index 000000000..26ecd9db3
--- /dev/null
+++ b/.sonarcloud.properties
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SonarCloud automatic analysis reads this file (not sonar-project.properties).
+# Multi-version Flink shims intentionally share parallel structure per Flink release.
+sonar.cpd.exclusions=**/streampark-flink-shims/**,**/streampark-flink-shims-base/**,**/streampark-flink-shims-base-v2/**,**/streampark-spark-shims/**
diff --git a/pom.xml b/pom.xml
index 0130fffb0..9f90652f9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -98,7 +98,7 @@
<flink.version>1.20.1</flink.version>
<flink.shaded.version>1.8.1</flink.shaded.version>
<streampark.shaded.version>1.0.0</streampark.shaded.version>
- <streampark.flink.shims.version>1.14</streampark.flink.shims.version>
+ <streampark.flink.shims.version>1.17</streampark.flink.shims.version>
<spark.version>3.1.2</spark.version>
<scalikejdbc.version>4.0.0</scalikejdbc.version>
<hive.version>2.3.4</hive.version>
@@ -222,7 +222,7 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<version>${project.version}</version>
</dependency>
@@ -932,6 +932,7 @@
<exclude>src/main/resources/*.dict</exclude>
<exclude>streampark-console-webapp/**</exclude>
+ <exclude>streampark-console-webapp-v2/**</exclude>
</excludes>
</configuration>
<executions>
diff --git a/streampark-common-scala-bridge/pom.xml b/streampark-common-scala-bridge/pom.xml
index 72daea88b..2344333ba 100644
--- a/streampark-common-scala-bridge/pom.xml
+++ b/streampark-common-scala-bridge/pom.xml
@@ -30,7 +30,7 @@
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
</dependency>
</dependencies>
diff --git a/streampark-common/pom.xml b/streampark-common/pom.xml
index 9659e267a..cba450c22 100644
--- a/streampark-common/pom.xml
+++ b/streampark-common/pom.xml
@@ -24,7 +24,7 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<name>StreamPark : Common</name>
<dependencies>
@@ -132,8 +132,8 @@
<plugins>
<plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
</plugin>
<plugin>
diff --git a/streampark-common/src/main/java/org/apache/streampark/common/conf/FlinkVersion.java b/streampark-common/src/main/java/org/apache/streampark/common/conf/FlinkVersion.java
index 394a8004e..aa1fd6553 100644
--- a/streampark-common/src/main/java/org/apache/streampark/common/conf/FlinkVersion.java
+++ b/streampark-common/src/main/java/org/apache/streampark/common/conf/FlinkVersion.java
@@ -28,6 +28,7 @@ import java.net.URL;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
+import java.util.function.Consumer;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -43,8 +44,11 @@ public class FlinkVersion implements Serializable {
private static final Pattern FLINK_VER_PATTERN = Pattern.compile("^(\\d+\\.\\d+)(?:\\..*)?$");
private static final Pattern FLINK_VERSION_PATTERN =
Pattern.compile("^Version: ([^,]*), Commit ID: (.*)$");
- private static final Pattern FLINK_SCALA_VERSION_PATTERN =
- Pattern.compile("^flink-dist_(\\d+\\.\\d+)[^/\\\\]*\\.jar$");
+ private static final Pattern FLINK_DIST_UNDERSCORE_PATTERN =
+ Pattern.compile(
+ "^flink-dist_(\\d+\\.\\d+)-(\\d+\\.\\d+(?:\\.\\d+)?(?:-SNAPSHOT)?)\\.jar$");
+ private static final Pattern FLINK_DIST_DASH_PATTERN =
+ Pattern.compile("^flink-dist-(\\d+\\.\\d+(?:\\.\\d+)?(?:-SNAPSHOT)?)\\.jar$");
private static final Pattern APACHE_FLINK_VERSION_PATTERN =
Pattern.compile("^(\\d+\\.\\d+\\.\\d+)");
private static final Pattern OTHER_FLINK_VERSION_PATTERN = Pattern.compile("^(\\d+\\.\\d+)-?$");
@@ -84,10 +88,14 @@ public class FlinkVersion implements Serializable {
return getVersion();
}
+ /** Backward-compatible alias for {@link #getFlinkLibs()}. */
+ public List<URL> flinkLibs() throws Exception {
+ return getFlinkLibs();
+ }
+
public String getScalaVersion() {
if (scalaVersion == null) {
- Matcher matcher = FLINK_SCALA_VERSION_PATTERN.matcher(getFlinkDistJar().getName());
- scalaVersion = matcher.matches() ? matcher.group(1) : "2.12";
+ scalaVersion = parseFromDistJar().map(pair -> pair[1]).orElse("2.12");
}
return scalaVersion;
}
@@ -130,41 +138,10 @@ public class FlinkVersion implements Serializable {
public String getVersion() {
if (version == null) {
- List<String> cmd =
- Arrays.asList(
- "java -classpath "
- + getFlinkDistJar().getName()
- + " org.apache.flink.client.cli.CliFrontend --version");
- StringBuilder buffer = new StringBuilder();
- final String[] flinkVersion = {null};
- try {
- CommandUtils.execute(
- getFlinkLib().getAbsolutePath(),
- cmd,
- out -> {
- buffer.append(out).append("\n");
- Matcher matcher = FLINK_VERSION_PATTERN.matcher(out);
- if (matcher.find()) {
- String ver = matcher.group(1);
- Matcher m1 = APACHE_FLINK_VERSION_PATTERN.matcher(ver);
- if (m1.find()) {
- flinkVersion[0] = ver;
- } else {
- Matcher m2 = OTHER_FLINK_VERSION_PATTERN.matcher(ver);
- if (m2.find()) {
- flinkVersion[0] = ver;
- }
- }
- }
- });
- } catch (Exception e) {
- throw new IllegalStateException("[StreamPark] execute flink version command failed", e);
- }
- LOG.info("[StreamPark] {}", buffer);
- if (flinkVersion[0] == null) {
- throw new IllegalStateException("[StreamPark] parse flink version failed. " + buffer);
- }
- version = flinkVersion[0];
+ version =
+ parseFromDistJar()
+ .map(pair -> pair[0])
+ .orElseGet(this::parseFromCliFrontend);
}
return version;
}
@@ -205,10 +182,14 @@ public class FlinkVersion implements Serializable {
public boolean checkVersion(boolean throwException) {
String[] parts = getVersion().split("\\.");
- if (parts.length >= 2 && "1".equals(parts[0].trim())) {
+ if (parts.length >= 2) {
try {
+ int major = Integer.parseInt(parts[0].trim());
int minor = Integer.parseInt(parts[1].trim());
- if (minor >= 12 && minor <= 20) {
+ if (major == 1 && minor >= 17 && minor <= 20) {
+ return true;
+ }
+ if (major == 2 && minor >= 0 && minor <= 3) {
return true;
}
} catch (NumberFormatException ignored) {
@@ -222,15 +203,88 @@ public class FlinkVersion implements Serializable {
public boolean checkVersion(int sinceVersion) {
String[] parts = getVersion().split("\\.");
- if (parts.length >= 2 && "1".equals(parts[0].trim())) {
+ if (parts.length >= 2) {
try {
- return Integer.parseInt(parts[1].trim()) >= sinceVersion;
+ int major = Integer.parseInt(parts[0].trim());
+ int minor = Integer.parseInt(parts[1].trim());
+ if (major == 1 && minor >= sinceVersion) {
+ return true;
+ }
+ if (major == 2) {
+ return true;
+ }
} catch (NumberFormatException ignored) {
}
}
return false;
}
+ private java.util.Optional<String[]> parseFromDistJar() {
+ String jarName = getFlinkDistJar().getName();
+ Matcher underscoreMatcher = FLINK_DIST_UNDERSCORE_PATTERN.matcher(jarName);
+ if (underscoreMatcher.matches()) {
+ String parsedVersion = underscoreMatcher.group(2);
+ String parsedScala = underscoreMatcher.group(1);
+ LOG.info(
+ "Flink version parsed from dist jar name: {}, scala: {}",
+ parsedVersion,
+ parsedScala);
+ return java.util.Optional.of(new String[]{parsedVersion, parsedScala});
+ }
+ Matcher dashMatcher = FLINK_DIST_DASH_PATTERN.matcher(jarName);
+ if (dashMatcher.matches()) {
+ String parsedVersion = dashMatcher.group(1);
+ LOG.info(
+ "Flink version parsed from dist jar name: {}, scala: {}",
+ parsedVersion,
+ "2.12");
+ return java.util.Optional.of(new String[]{parsedVersion, "2.12"});
+ }
+ return java.util.Optional.empty();
+ }
+
+ private String parseFromCliFrontend() {
+ final String[] flinkVersion = {null};
+ StringBuilder buffer = new StringBuilder();
+ List<String> cmd =
+ Arrays.asList(
+ "java -classpath "
+ + getFlinkDistJar().getName()
+ + " org.apache.flink.client.cli.CliFrontend --version");
+ try {
+ CommandUtils.execute(
+ getFlinkLib().getAbsolutePath(),
+ cmd,
+ new Consumer<String>() {
+
+ @Override
+ public void accept(String out) {
+ buffer.append(out).append("\n");
+ Matcher matcher = FLINK_VERSION_PATTERN.matcher(out);
+ if (matcher.find()) {
+ String ver = matcher.group(1);
+ Matcher m1 = APACHE_FLINK_VERSION_PATTERN.matcher(ver);
+ if (m1.find()) {
+ flinkVersion[0] = ver;
+ } else {
+ Matcher m2 = OTHER_FLINK_VERSION_PATTERN.matcher(ver);
+ if (m2.find()) {
+ flinkVersion[0] = ver;
+ }
+ }
+ }
+ }
+ });
+ } catch (Exception e) {
+ throw new IllegalStateException("[StreamPark] execute flink version command failed", e);
+ }
+ LOG.info("[StreamPark] {}", buffer);
+ if (flinkVersion[0] == null) {
+ throw new IllegalStateException("[StreamPark] parse flink version failed. " + buffer);
+ }
+ return flinkVersion[0];
+ }
+
@Override
public String toString() {
return "\n----------------------------------------- flink version -----------------------------------\n"
diff --git a/streampark-console/streampark-console-service/pom.xml b/streampark-console/streampark-console-service/pom.xml
index 3d8431b4d..139433519 100644
--- a/streampark-console/streampark-console-service/pom.xml
+++ b/streampark-console/streampark-console-service/pom.xml
@@ -35,7 +35,7 @@
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<postgresql.version>42.5.1</postgresql.version>
<mybatis-plus.version>3.5.3.1</mybatis-plus.version>
- <streampark.flink.shims.version>1.14</streampark.flink.shims.version>
+ <streampark.flink.shims.version>1.17</streampark.flink.shims.version>
<frontend.project.name>streampark-console-webapp</frontend.project.name>
<CodeCacheSize>512m</CodeCacheSize>
@@ -72,6 +72,7 @@
</dependencyManagement>
<dependencies>
+
<!--scala-->
<dependency>
<groupId>org.scala-lang</groupId>
@@ -357,19 +358,76 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims-base_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-base</artifactId>
<version>${project.version}</version>
</dependency>
+ <!-- Ensure all Flink shims are built when console-service is built with -am -->
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-1.17</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-1.18</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-1.19</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-1.20</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-2.0</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-2.1</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-shims_flink-2.2</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-client-api_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-2.3</artifactId>
<version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-client-api</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-client-core</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
@@ -394,7 +452,14 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>streampark-spark-client-core_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-flink-kubernetes</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
@@ -406,7 +471,7 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-sqlclient_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-sqlclient</artifactId>
<version>${project.version}</version>
</dependency>
@@ -416,11 +481,6 @@
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.module</groupId>
- <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
- </dependency>
-
<!--Test dependencies start.-->
<dependency>
<groupId>org.assertj</groupId>
@@ -529,73 +589,66 @@
<excludeTransitive>false</excludeTransitive>
<stripVersion>false</stripVersion>
<artifactItems>
- <!-- flink 1.12 support-->
- <dependency>
- <groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.12_${scala.binary.version}</artifactId>
- <version>${project.version}</version>
- <outputDirectory>${project.build.directory}/shims</outputDirectory>
- </dependency>
- <!-- flink 1.13 support-->
+ <!-- flink 1.17 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.13_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-1.17</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.14 support-->
+ <!-- flink 1.18 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.14_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-1.18</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.15 support-->
+ <!-- flink 1.19 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.15_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-1.19</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.16 support-->
+ <!-- flink 1.20 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.16_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-1.20</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.17 support-->
+ <!-- flink 2.0 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.17_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-2.0</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.18 support-->
+ <!-- flink 2.1 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.18_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-2.1</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.19 support-->
+ <!-- flink 2.2 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.19_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-2.2</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
- <!-- flink 1.20 support-->
+ <!-- flink 2.3 support-->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.20_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-2.3</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/shims</outputDirectory>
</dependency>
<!-- flink-submit-core -->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-client-core_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-client-core</artifactId>
<version>${project.version}</version>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
</dependency>
@@ -643,21 +696,6 @@
</executions>
</plugin>
- <plugin>
- <groupId>com.diffplug.spotless</groupId>
- <artifactId>spotless-maven-plugin</artifactId>
- <version>${maven-spotless-plugin.version}</version>
- <executions>
- <execution>
- <id>spotless-check</id>
- <goals>
- <goal>check</goal>
- </goals>
- <phase>validate</phase>
- </execution>
- </executions>
- </plugin>
-
</plugins>
<sourceDirectory>src/main/java</sourceDirectory>
diff --git a/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/runner/EnvInitializer.java b/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/runner/EnvInitializer.java
index c9addb417..6ab518f6a 100644
--- a/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/runner/EnvInitializer.java
+++ b/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/runner/EnvInitializer.java
@@ -71,7 +71,7 @@ public class EnvInitializer implements ApplicationRunner {
private final FileFilter fileFilter = p -> !".gitkeep".equals(p.getName());
private static final Pattern PATTERN_FLINK_SHIMS_JAR = Pattern.compile(
- "^streampark-flink-shims_flink-(1.1[2-9]|1\\.2[0-9])_(2.12)-(.*).jar$",
+ "^streampark-flink-shims_flink-(1\\.1[7-9]|1\\.2[0-9]|2\\.[0-3])-(.*).jar$",
Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
@SneakyThrows
diff --git a/streampark-e2e/streampark-e2e-case/src/test/java/org/apache/streampark/e2e/cases/ProjectsManagementTest.java b/streampark-e2e/streampark-e2e-case/src/test/java/org/apache/streampark/e2e/cases/ProjectsManagementTest.java
index 068f25ebb..d6b1b82f6 100644
--- a/streampark-e2e/streampark-e2e-case/src/test/java/org/apache/streampark/e2e/cases/ProjectsManagementTest.java
+++ b/streampark-e2e/streampark-e2e-case/src/test/java/org/apache/streampark/e2e/cases/ProjectsManagementTest.java
@@ -50,7 +50,7 @@ public class ProjectsManagementTest {
private static final String branch = "dev";
private static final String buildArgument =
- "-pl quickstart-flink/quickstart-apacheflink/apacheflinksql_1.16 -am -Dmaven.test.skip=true";
+ "-pl quickstart-flink/quickstart-apacheflink/apacheflinksql_1.17 -am -Dmaven.test.skip=true";
private static final String description = "e2e test project description";
diff --git a/streampark-flink/pom.xml b/streampark-flink/pom.xml
index e4f4ffef6..c84c942ab 100644
--- a/streampark-flink/pom.xml
+++ b/streampark-flink/pom.xml
@@ -65,4 +65,13 @@
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <id>connector</id>
+ <modules>
+ <module>streampark-flink-connector</module>
+ </modules>
+ </profile>
+ </profiles>
+
</project>
diff --git a/streampark-flink/streampark-flink-client/streampark-flink-client-api/pom.xml b/streampark-flink/streampark-flink-client/streampark-flink-client-api/pom.xml
index bb2f5c371..a3442949d 100644
--- a/streampark-flink/streampark-flink-client/streampark-flink-client-api/pom.xml
+++ b/streampark-flink/streampark-flink-client/streampark-flink-client-api/pom.xml
@@ -24,39 +24,39 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-client-api_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-client-api</artifactId>
<name>StreamPark : Flink Client Api</name>
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-proxy_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-proxy</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-packer_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-packer</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-kubernetes</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims-base_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-base</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
@@ -72,24 +72,6 @@
<artifactId>hadoop-client-runtime</artifactId>
<scope>provided</scope>
</dependency>
-
- <dependency>
- <groupId>org.projectlombok</groupId>
- <artifactId>lombok</artifactId>
- <scope>provided</scope>
- </dependency>
-
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.assertj</groupId>
- <artifactId>assertj-core</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<profiles>
diff --git a/streampark-flink/streampark-flink-client/streampark-flink-client-core/pom.xml b/streampark-flink/streampark-flink-client/streampark-flink-client-core/pom.xml
index 1b50fc9e0..d0f43da7e 100644
--- a/streampark-flink/streampark-flink-client/streampark-flink-client-core/pom.xml
+++ b/streampark-flink/streampark-flink-client/streampark-flink-client-core/pom.xml
@@ -24,30 +24,29 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-client-core_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-client-core</artifactId>
<name>StreamPark : Flink Client Core</name>
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-client-api_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-client-api</artifactId>
<version>${project.version}</version>
- <scope>provided</scope>
</dependency>
<!-- The flink session mode would resolve the DAG within StreamPark process,
so a full build dependencies is required. -->
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-${streampark.flink.shims.version}_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-${streampark.flink.shims.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-kubernetes</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
@@ -89,34 +88,7 @@
<artifactId>guava</artifactId>
<scope>provided</scope>
</dependency>
-
- <dependency>
- <groupId>org.projectlombok</groupId>
- <artifactId>lombok</artifactId>
- <scope>provided</scope>
- </dependency>
-
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.assertj</groupId>
- <artifactId>assertj-core</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>build-helper-maven-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
<profiles>
<profile>
<id>apache-release</id>
diff --git a/streampark-flink/streampark-flink-client/streampark-flink-client-core/src/test/java/org/apache/streampark/flink/client/test/YarnPerJobTestCase.java b/streampark-flink/streampark-flink-client/streampark-flink-client-core/src/test/java/org/apache/streampark/flink/client/test/YarnPerJobTestCase.java
new file mode 100644
index 000000000..51950d525
--- /dev/null
+++ b/streampark-flink/streampark-flink-client/streampark-flink-client-core/src/test/java/org/apache/streampark/flink/client/test/YarnPerJobTestCase.java
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.client.test;
+
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
+import org.apache.streampark.flink.client.bean.SubmitResponse;
+import org.apache.streampark.flink.core.conf.FlinkRunOption;
+
+import org.apache.streampark.shaded.org.slf4j.Logger;
+
+import org.apache.commons.cli.Options;
+import org.apache.flink.client.cli.CliFrontend;
+import org.apache.flink.client.cli.CliFrontendParser;
+import org.apache.flink.client.cli.CustomCommandLine;
+import org.apache.flink.client.deployment.ClusterSpecification;
+import org.apache.flink.client.deployment.DefaultClusterClientServiceLoader;
+import org.apache.flink.client.program.ClusterClientProvider;
+import org.apache.flink.client.program.PackagedProgram;
+import org.apache.flink.client.program.PackagedProgramUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.configuration.GlobalConfiguration;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.MemorySize.MemoryUnit;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.yarn.YarnClusterDescriptor;
+import org.apache.flink.yarn.configuration.YarnDeploymentTarget;
+import org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** perJob to submit jobs programmatically */
+public final class YarnPerJobTestCase {
+
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory().getLogger(YarnPerJobTestCase.class.getName());
+
+ private static final String PROGRAM_ARGS = "--hostname localhost --port 9999";
+
+ private static final String OPTION = "-e yarn-per-job -p 2 -n";
+
+ private static Configuration flinkDefaultConfiguration;
+ private static List<CustomCommandLine> customCommandLines;
+ private static Method deployInternalMethod;
+
+ private YarnPerJobTestCase() {
+ }
+
+ private static void ensureInitialized() {
+ if (deployInternalMethod != null) {
+ return;
+ }
+ String flinkHome = Objects.requireNonNull(System.getenv("FLINK_HOME"), "FLINK_HOME must be set");
+ LOG.info("flinkHome: {}", flinkHome);
+ flinkDefaultConfiguration = GlobalConfiguration.loadConfiguration(flinkHome + "/conf");
+ try {
+ customCommandLines =
+ CliFrontend.loadCustomCommandLines(flinkDefaultConfiguration, flinkHome + "/conf");
+ Class<?>[] paramClass =
+ new Class<?>[]{
+ ClusterSpecification.class,
+ String.class,
+ String.class,
+ JobGraph.class,
+ boolean.class
+ };
+ deployInternalMethod =
+ YarnClusterDescriptor.class.getDeclaredMethod("deployInternal", paramClass);
+ deployInternalMethod.setAccessible(true);
+ } catch (Exception e) {
+ throw new IllegalStateException("Failed to initialize YARN integration harness", e);
+ }
+ }
+
+ @Test
+ void verifyYarnPerJobHarnessCompatibility() throws Exception {
+ Method deployInternal =
+ YarnClusterDescriptor.class.getDeclaredMethod(
+ "deployInternal",
+ ClusterSpecification.class,
+ String.class,
+ String.class,
+ JobGraph.class,
+ boolean.class);
+ assertThat(deployInternal).isNotNull();
+
+ SubmitResponse response = new SubmitResponse("application_123", Collections.emptyMap());
+ assertThat(response.clusterId()).isEqualTo("application_123");
+ assertThat(YarnDeploymentTarget.PER_JOB.getName()).isEqualTo("yarn-per-job");
+
+ Options commandLineOptions =
+ FlinkRunOption.mergeOptions(FlinkRunOption.getRunCommandOptions(), new Options());
+ org.apache.commons.cli.CommandLine commandLine =
+ FlinkRunOption.parse(commandLineOptions, OPTION.split("\\s+"), true);
+ assertThat(commandLine.getOptionValue("e")).isEqualTo("yarn-per-job");
+
+ String flinkHome = System.getenv("FLINK_HOME");
+ if (flinkHome != null) {
+ ensureInitialized();
+ assertThat(customCommandLines).isNotEmpty();
+ assertThat(new File(flinkHome, "lib").exists()).isTrue();
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private static ClusterClientProvider<ApplicationId> deployInternal(
+ YarnClusterDescriptor clusterDescriptor,
+ ClusterSpecification clusterSpecification,
+ String applicationName,
+ String yarnClusterEntrypoint,
+ JobGraph jobGraph,
+ Boolean detached) throws Exception {
+ return (ClusterClientProvider<ApplicationId>) deployInternalMethod.invoke(
+ clusterDescriptor,
+ clusterSpecification,
+ applicationName,
+ yarnClusterEntrypoint,
+ jobGraph,
+ detached);
+ }
+
+ public static void main(String[] args) throws Exception {
+ ensureInitialized();
+ String flinkHome = System.getenv("FLINK_HOME");
+ String userJar = flinkHome + "/examples/streaming/SocketWindowWordCount.jar";
+ Options customCommandLineOptions = new Options();
+ for (CustomCommandLine customCommandLine : customCommandLines) {
+ customCommandLine.addGeneralOptions(customCommandLineOptions);
+ customCommandLine.addRunOptions(customCommandLineOptions);
+ }
+ Options commandLineOptions =
+ FlinkRunOption.mergeOptions(
+ CliFrontendParser.getRunCommandOptions(), customCommandLineOptions);
+ org.apache.commons.cli.CommandLine commandLine =
+ FlinkRunOption.parse(commandLineOptions, OPTION.split("\\s+"), true);
+
+ CustomCommandLine activeCommandLine = null;
+ LOG.info("Custom commandlines: {}", customCommandLines);
+ for (CustomCommandLine cli : customCommandLines) {
+ if (cli.isActive(Preconditions.checkNotNull(commandLine))) {
+ activeCommandLine = cli;
+ break;
+ }
+ }
+ if (activeCommandLine == null) {
+ throw new IllegalStateException("No valid command-line found.");
+ }
+
+ Configuration executorConfig = activeCommandLine.toConfiguration(commandLine);
+ Configuration flinkConfig = new Configuration(executorConfig);
+ flinkConfig.set(DeploymentOptions.TARGET, YarnDeploymentTarget.PER_JOB.getName());
+ flinkConfig.set(
+ org.apache.flink.client.deployment.application.ApplicationConfiguration.APPLICATION_ARGS,
+ Arrays.asList(PROGRAM_ARGS.split("\\s+")));
+ flinkConfig.set(
+ JobManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1024", MemoryUnit.MEGA_BYTES));
+ flinkConfig.set(
+ TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1024", MemoryUnit.MEGA_BYTES));
+
+ DefaultClusterClientServiceLoader clusterClientServiceLoader =
+ new DefaultClusterClientServiceLoader();
+ org.apache.flink.client.deployment.ClusterClientFactory<ApplicationId> clientFactory =
+ clusterClientServiceLoader.getClusterClientFactory(flinkConfig);
+
+ YarnClusterDescriptor clusterDescriptor =
+ (YarnClusterDescriptor) clientFactory.createClusterDescriptor(flinkConfig);
+ String[] distJars =
+ new File(flinkHome + "/lib")
+ .list((dir, name) -> name.matches("flink-dist.*\\.jar"));
+ if (distJars == null || distJars.length == 0) {
+ throw new IllegalArgumentException(
+ "[StreamPark] can no found flink-dist jar in " + flinkHome + "/lib");
+ }
+ if (distJars.length > 1) {
+ throw new IllegalArgumentException(
+ "[StreamPark] found multiple flink-dist jar in "
+ + flinkHome
+ + "/lib,["
+ + String.join(",", distJars)
+ + "]");
+ }
+ clusterDescriptor.setLocalJarPath(new Path(flinkHome + "/lib/" + distJars[0]));
+
+ try {
+ ClusterSpecification clusterSpecification =
+ clientFactory.getClusterSpecification(flinkConfig);
+ LOG.info("------------------<<specification>>------------------");
+ LOG.info("{}", clusterSpecification);
+ LOG.info("------------------------------------");
+
+ PackagedProgram packagedProgram =
+ PackagedProgram.newBuilder()
+ .setJarFile(new File(userJar))
+ .setArguments(PROGRAM_ARGS.split("\\s+"))
+ .build();
+ JobGraph jobGraph =
+ PackagedProgramUtils.createJobGraph(packagedProgram, flinkConfig, 1, false);
+ LOG.info("------------------<<jobId>>------------------");
+ LOG.info("{}", jobGraph.getJobID());
+ LOG.info("------------------------------------");
+
+ org.apache.flink.client.program.ClusterClient<ApplicationId> clusterClient =
+ deployInternal(
+ clusterDescriptor,
+ clusterSpecification,
+ "MyJob",
+ YarnJobClusterEntrypoint.class.getName(),
+ jobGraph,
+ false)
+ .getClusterClient();
+ ApplicationId applicationId = clusterClient.getClusterId();
+ LOG.info("------------------<<applicationId>>-------------------");
+ LOG.info("Flink Job Started: applicationId: {} ", applicationId);
+ LOG.info("-------------------------------------");
+ new SubmitResponse(applicationId.toString(), flinkConfig.toMap());
+ } finally {
+ clusterDescriptor.close();
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-kubernetes/pom.xml b/streampark-flink/streampark-flink-kubernetes/pom.xml
index aab79be6d..1cc45a167 100644
--- a/streampark-flink/streampark-flink-kubernetes/pom.xml
+++ b/streampark-flink/streampark-flink-kubernetes/pom.xml
@@ -25,7 +25,7 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-kubernetes</artifactId>
<name>StreamPark : Flink Kubernetes Integration</name>
<properties>
@@ -89,14 +89,14 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-${streampark.flink.shims.version}_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-${streampark.flink.shims.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
@@ -162,12 +162,6 @@
<artifactId>lombok</artifactId>
</dependency>
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- <scope>test</scope>
- </dependency>
-
</dependencies>
<build>
diff --git a/streampark-flink/streampark-flink-kubernetes/src/main/java/org/apache/streampark/flink/kubernetes/watcher/FlinkRestModels.java b/streampark-flink/streampark-flink-kubernetes/src/main/java/org/apache/streampark/flink/kubernetes/watcher/FlinkRestModels.java
new file mode 100644
index 000000000..2e115b466
--- /dev/null
+++ b/streampark-flink/streampark-flink-kubernetes/src/main/java/org/apache/streampark/flink/kubernetes/watcher/FlinkRestModels.java
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.kubernetes.watcher;
+
+import org.apache.streampark.common.util.JsonUtils;
+import org.apache.streampark.flink.kubernetes.enums.FlinkJobState;
+import org.apache.streampark.flink.kubernetes.model.JobStatusCV;
+
+import org.apache.streampark.shaded.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+/** Flink REST API response models and parsers. */
+final class FlinkRestModels {
+
+ private FlinkRestModels() {
+ }
+
+ static Optional<JobDetails> parseJobDetails(String json) {
+ try {
+ JsonNode root = JsonUtils.read(json, JsonNode.class);
+ JsonNode jobsNode = root.get("jobs");
+ if (jobsNode == null || jobsNode.isNull() || !jobsNode.isArray()) {
+ return Optional.empty();
+ }
+ List<JobDetail> details = new ArrayList<>();
+ for (JsonNode node : jobsNode) {
+ details.add(new JobDetail(node));
+ }
+ return Optional.of(new JobDetails(details.toArray(new JobDetail[0])));
+ } catch (Exception e) {
+ return Optional.empty();
+ }
+ }
+
+ static Optional<FlinkRestOverview> parseOverview(String json) {
+ try {
+ JsonNode root = JsonUtils.read(json, JsonNode.class);
+ return Optional.of(new FlinkRestOverview(root));
+ } catch (Exception e) {
+ return Optional.empty();
+ }
+ }
+
+ static List<FlinkRestJmConfigItem> parseJmConfig(String json) {
+ try {
+ JsonNode root = JsonUtils.read(json, JsonNode.class);
+ if (!root.isArray()) {
+ return Collections.emptyList();
+ }
+ List<FlinkRestJmConfigItem> items = new ArrayList<>();
+ for (JsonNode node : root) {
+ items.add(new FlinkRestJmConfigItem(text(node, "key"), text(node, "value")));
+ }
+ return items;
+ } catch (Exception e) {
+ return Collections.emptyList();
+ }
+ }
+
+ static Optional<CheckpointResponse> parseCheckpoint(String json) {
+ try {
+ JsonNode root = JsonUtils.read(json, JsonNode.class);
+ JsonNode completed = root.path("latest").path("completed");
+ if (completed.isMissingNode() || completed.isNull()) {
+ return Optional.empty();
+ }
+ return Optional.of(new CheckpointResponse(completed));
+ } catch (Exception e) {
+ return Optional.empty();
+ }
+ }
+
+ private static String text(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ return value == null || value.isNull() ? null : value.asText();
+ }
+
+ private static int intVal(JsonNode node, String field) {
+ JsonNode value = node == null ? null : node.get(field);
+ return value == null || value.isNull() ? 0 : value.asInt();
+ }
+
+ private static long longVal(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ return value == null || value.isNull() ? 0L : value.asLong();
+ }
+
+ static final class JobDetails {
+
+ private final JobDetail[] jobs;
+
+ JobDetails(JobDetail[] jobs) {
+ this.jobs = jobs;
+ }
+
+ JobDetail[] jobs() {
+ return jobs;
+ }
+ }
+
+ static final class JobDetail {
+
+ private final String jid;
+ private final String name;
+ private final String state;
+ private final long startTime;
+ private final long endTime;
+ private final long duration;
+ private final long lastModification;
+ private final JobTask tasks;
+
+ JobDetail(JsonNode node) {
+ this.jid = text(node, "jid");
+ this.name = text(node, "name");
+ this.state = text(node, "state");
+ this.startTime = longVal(node, "start-time");
+ this.endTime = longVal(node, "end-time");
+ this.duration = longVal(node, "duration");
+ this.lastModification = longVal(node, "last-modification");
+ this.tasks = new JobTask(node.get("tasks"));
+ }
+
+ JobStatusCV toJobStatusCV(long pollEmitTime, long pollAckTime) {
+ return new JobStatusCV(
+ FlinkJobState.of(state),
+ jid,
+ name,
+ startTime,
+ endTime,
+ duration,
+ tasks.total,
+ pollEmitTime,
+ pollAckTime);
+ }
+
+ String jid() {
+ return jid;
+ }
+ }
+
+ static final class JobTask {
+
+ private final int total;
+ private final int created;
+ private final int scheduled;
+ private final int deploying;
+ private final int running;
+ private final int finished;
+ private final int canceling;
+ private final int canceled;
+ private final int failed;
+ private final int reconciling;
+ private final int initializing;
+
+ JobTask(JsonNode task) {
+ this.total = intVal(task, "total");
+ this.created = intVal(task, "created");
+ this.scheduled = intVal(task, "scheduled");
+ this.deploying = intVal(task, "deploying");
+ this.running = intVal(task, "running");
+ this.finished = intVal(task, "finished");
+ this.canceling = intVal(task, "canceling");
+ this.canceled = intVal(task, "canceled");
+ this.failed = intVal(task, "failed");
+ this.reconciling = intVal(task, "reconciling");
+ this.initializing = intVal(task, "initializing");
+ }
+ }
+
+ static final class FlinkRestOverview {
+
+ private final Integer taskManagers;
+ private final Integer slotsTotal;
+ private final Integer slotsAvailable;
+ private final Integer jobsRunning;
+ private final Integer jobsFinished;
+ private final Integer jobsCancelled;
+ private final Integer jobsFailed;
+ private final String flinkVersion;
+
+ FlinkRestOverview(JsonNode root) {
+ this.taskManagers = intVal(root, "taskmanagers");
+ this.slotsTotal = intVal(root, "slots-total");
+ this.slotsAvailable = intVal(root, "slots-available");
+ this.jobsRunning = intVal(root, "jobs-running");
+ this.jobsFinished = intVal(root, "jobs-finished");
+ this.jobsCancelled = intVal(root, "jobs-cancelled");
+ this.jobsFailed = intVal(root, "jobs-failed");
+ this.flinkVersion = text(root, "flink-version");
+ }
+
+ Integer taskManagers() {
+ return taskManagers;
+ }
+ Integer slotsTotal() {
+ return slotsTotal;
+ }
+ Integer slotsAvailable() {
+ return slotsAvailable;
+ }
+ Integer jobsRunning() {
+ return jobsRunning;
+ }
+ Integer jobsFinished() {
+ return jobsFinished;
+ }
+ Integer jobsCancelled() {
+ return jobsCancelled;
+ }
+ Integer jobsFailed() {
+ return jobsFailed;
+ }
+ }
+
+ static final class FlinkRestJmConfigItem {
+
+ private final String key;
+ private final String value;
+
+ FlinkRestJmConfigItem(String key, String value) {
+ this.key = key;
+ this.value = value;
+ }
+
+ String key() {
+ return key;
+ }
+
+ String value() {
+ return value;
+ }
+ }
+
+ static final class CheckpointResponse {
+
+ private final long id;
+ private final String status;
+ private final String externalPath;
+ private final boolean isSavepoint;
+ private final String checkpointType;
+ private final long triggerTimestamp;
+
+ CheckpointResponse(JsonNode completed) {
+ this.id = longVal(completed, "id");
+ this.status = text(completed, "status");
+ this.externalPath = text(completed, "external_path");
+ this.isSavepoint = completed.path("is_savepoint").asBoolean(false);
+ this.checkpointType = text(completed, "checkpoint_type");
+ this.triggerTimestamp = longVal(completed, "trigger_timestamp");
+ }
+
+ long id() {
+ return id;
+ }
+ String status() {
+ return status;
+ }
+ String externalPath() {
+ return externalPath;
+ }
+ boolean isSavepoint() {
+ return isSavepoint;
+ }
+ String checkpointType() {
+ return checkpointType;
+ }
+ long triggerTimestamp() {
+ return triggerTimestamp;
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-packer/pom.xml b/streampark-flink/streampark-flink-packer/pom.xml
index bb3adf636..b414426bf 100644
--- a/streampark-flink/streampark-flink-packer/pom.xml
+++ b/streampark-flink/streampark-flink-packer/pom.xml
@@ -24,13 +24,13 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-packer_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-packer</artifactId>
<name>StreamPark : Flink Packer</name>
<properties>
<eclipse.aether.version>1.1.0</eclipse.aether.version>
<maven.aether.version>3.3.9</maven.aether.version>
- <maven.shade.version>${maven-shade-plugin.version}</maven.shade.version>
+ <maven.shade.version>3.2.4</maven.shade.version>
<docker.client.version>3.3.6</docker.client.version>
<httpclient.version>4.5.14</httpclient.version>
</properties>
@@ -39,13 +39,13 @@
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-kubernetes</artifactId>
<version>${project.version}</version>
</dependency>
@@ -136,11 +136,16 @@
<artifactId>guava</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ </dependency>
+
<!-- jackson-->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
- <version>${jackson.version}</version>
+ <version>2.12.0</version>
</dependency>
<dependency>
diff --git a/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/docker/DockerClients.java b/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/docker/DockerClients.java
new file mode 100644
index 000000000..f5af23e31
--- /dev/null
+++ b/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/docker/DockerClients.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.packer.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.listener.BuildImageCallbackListener;
+import com.github.dockerjava.api.listener.PullImageCallbackListener;
+import com.github.dockerjava.api.listener.PushImageCallbackListener;
+import com.github.dockerjava.api.model.PullResponseItem;
+import com.github.dockerjava.api.model.PushResponseItem;
+
+import java.util.function.Consumer;
+import java.util.function.Function;
+
+public final class DockerClients {
+
+ private DockerClients() {
+ }
+
+ public static BuildImageCallbackListener watchDockerBuildStep(Consumer<String> func) {
+ return new BuildImageCallbackListener() {
+
+ @Override
+ public void watchBuildStep(String buildStepMsg) {
+ func.accept(buildStepMsg);
+ }
+ };
+ }
+
+ public static PullImageCallbackListener watchDockerPullProcess(Consumer<PullResponseItem> func) {
+ return new PullImageCallbackListener() {
+
+ @Override
+ public void watchPullProcess(PullResponseItem processDetail) {
+ func.accept(processDetail);
+ }
+ };
+ }
+
+ public static PushImageCallbackListener watchDockerPushProcess(Consumer<PushResponseItem> func) {
+ return new PushImageCallbackListener() {
+
+ @Override
+ public void watchPushProcess(PushResponseItem processDetail) {
+ func.accept(processDetail);
+ }
+ };
+ }
+
+ public static <R> R usingDockerClient(Function<DockerClient, R> process, Function<Throwable, R> handleException) {
+ try {
+ DockerClient client = DockerRetriever.newDockerClient();
+ try {
+ return process.apply(client);
+ } finally {
+ client.close();
+ }
+ } catch (Throwable e) {
+ return handleException.apply(e);
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-proxy/pom.xml b/streampark-flink/streampark-flink-proxy/pom.xml
index 888754ba5..b056e1b24 100644
--- a/streampark-flink/streampark-flink-proxy/pom.xml
+++ b/streampark-flink/streampark-flink-proxy/pom.xml
@@ -24,23 +24,17 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-proxy_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-proxy</artifactId>
<name>StreamPark : Flink Proxy</name>
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- <scope>test</scope>
- </dependency>
-
</dependencies>
<profiles>
diff --git a/streampark-flink/streampark-flink-shims/pom.xml b/streampark-flink/streampark-flink-shims/pom.xml
index 734e6efa2..42282c463 100644
--- a/streampark-flink/streampark-flink-shims/pom.xml
+++ b/streampark-flink/streampark-flink-shims/pom.xml
@@ -31,16 +31,28 @@
<modules>
<module>streampark-flink-shims-base</module>
+ <module>streampark-flink-shims-base-v2</module>
<module>streampark-flink-shims-test</module>
- <module>streampark-flink-shims_flink-1.12</module>
- <module>streampark-flink-shims_flink-1.13</module>
- <module>streampark-flink-shims_flink-1.14</module>
- <module>streampark-flink-shims_flink-1.15</module>
- <module>streampark-flink-shims_flink-1.16</module>
<module>streampark-flink-shims_flink-1.17</module>
<module>streampark-flink-shims_flink-1.18</module>
<module>streampark-flink-shims_flink-1.19</module>
<module>streampark-flink-shims_flink-1.20</module>
</modules>
+ <profiles>
+ <!-- Flink 2.x APIs require JDK 11+ to compile against -->
+ <profile>
+ <id>flink-2.x-shims</id>
+ <activation>
+ <jdk>[11,)</jdk>
+ </activation>
+ <modules>
+ <module>streampark-flink-shims_flink-2.0</module>
+ <module>streampark-flink-shims_flink-2.1</module>
+ <module>streampark-flink-shims_flink-2.2</module>
+ <module>streampark-flink-shims_flink-2.3</module>
+ </modules>
+ </profile>
+ </profiles>
+
</project>
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/pom.xml b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/pom.xml
similarity index 70%
rename from streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/pom.xml
rename to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/pom.xml
index 7a6501ec4..3d4af1042 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/pom.xml
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/pom.xml
@@ -17,7 +17,6 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
-
<modelVersion>4.0.0</modelVersion>
<parent>
@@ -26,69 +25,85 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-shims_flink-1.12_${scala.binary.version}</artifactId>
- <name>StreamPark : Flink Shims 1.12</name>
+ <artifactId>streampark-flink-shims-base-v2</artifactId>
+ <name>StreamPark : Flink Shims Base V2</name>
<properties>
- <flink.version>1.12.5</flink.version>
+ <flink.version>2.0.2</flink.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims-base_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-base</artifactId>
<version>${project.version}</version>
</dependency>
- <!--flink-->
+ <dependency>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-common</artifactId>
+ </dependency>
+
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala_${scala.binary.version}</artifactId>
+ <artifactId>flink-core</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-scala_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
+ <artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-uber-blink_${scala.binary.version}</artifactId>
+ <artifactId>flink-kubernetes</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-statebackend-rocksdb_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-kubernetes_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ <optional>true</optional>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ <optional>true</optional>
+ </dependency>
</dependencies>
</project>
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
similarity index 60%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
index 5ed66ed7a..f5c53e692 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
@@ -18,45 +18,60 @@
package org.apache.streampark.flink.core;
import org.apache.streampark.common.conf.ConfigKeys;
+import org.apache.streampark.common.util.AssertUtils;
import org.apache.streampark.common.util.DeflaterUtils;
-import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.PipelineOptions;
import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.util.ParameterTool;
-public final class FlinkEnvironmentUtils {
+import javax.annotation.Nullable;
- private FlinkEnvironmentUtils() {
+/** Parameter and table environment helpers for Flink application naming. */
+public final class FlinkParameterUtils {
+
+ private FlinkParameterUtils() {
}
public static String getAppName(ParameterTool parameterTool) {
return getAppName(parameterTool, null, false);
}
- public static String getAppName(ParameterTool parameterTool, String name, boolean required) {
- String appName = name;
- if (appName == null) {
- try {
- appName =
- DeflaterUtils.unzipString(parameterTool.get(ConfigKeys.KEY_APP_NAME(), null));
- } catch (Exception ignored) {
- // match Scala Try(...).getOrElse(...)
+ public static String getAppName(ParameterTool parameterTool, boolean required) {
+ return getAppName(parameterTool, null, required);
+ }
+
+ public static String getAppName(
+ ParameterTool parameterTool, @Nullable String name, boolean required) {
+ String appName;
+ if (name == null) {
+ appName = null;
+ String zippedAppName = parameterTool.get(ConfigKeys.KEY_APP_NAME(), null);
+ if (zippedAppName != null) {
+ try {
+ appName = DeflaterUtils.unzipString(zippedAppName);
+ } catch (Exception ignored) {
+ // fall back to pipeline.name
+ }
}
if (appName == null) {
appName = parameterTool.get(ConfigKeys.KEY_FLINK_APP_NAME(), null);
}
+ } else {
+ appName = name;
}
- if (required && appName == null) {
- throw new IllegalArgumentException("[StreamPark] Application name cannot be null");
+ if (required) {
+ AssertUtils.required(
+ appName != null, "[StreamPark] Application name cannot be null");
}
return appName;
}
- public static TableEnvironment setAppName(TableEnvironment env, ParameterTool parameter) {
+ public static <T extends TableEnvironment> T setAppName(T env, ParameterTool parameter) {
String appName = getAppName(parameter);
if (appName != null) {
- env.getConfig().getConfiguration().setString(PipelineOptions.NAME, appName);
+ env.getConfig().getConfiguration().setString(PipelineOptions.NAME.key(), appName);
}
return env;
}
@@ -65,7 +80,7 @@ public final class FlinkEnvironmentUtils {
StreamTableEnvironment env, ParameterTool parameter) {
String appName = getAppName(parameter);
if (appName != null) {
- env.getConfig().getConfiguration().setString(PipelineOptions.NAME, appName);
+ env.getConfig().getConfiguration().setString(PipelineOptions.NAME.key(), appName);
}
return env;
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java
new file mode 100644
index 000000000..eae3584f5
--- /dev/null
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.core;
+
+import org.apache.streampark.common.conf.ConfigKeys;
+import org.apache.streampark.common.util.AssertUtils;
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
+
+import org.apache.streampark.shaded.org.slf4j.Logger;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ExecutionOptions;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.ParameterTool;
+
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/** Executes Flink SQL statements. */
+public final class FlinkSqlExecutor {
+
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory()
+ .getLogger(FlinkSqlExecutor.class.getName());
+
+ private static final ReentrantReadWriteLock.WriteLock LOCK =
+ new ReentrantReadWriteLock().writeLock();
+
+ private static final Map<SqlCommand, CommandHandler> COMMAND_HANDLERS = buildCommandHandlers();
+
+ private FlinkSqlExecutor() {
+ }
+
+ public static void executeSql(String sql, ParameterTool parameter, TableEnvironment context) {
+ executeSql(sql, parameter, context, null);
+ }
+
+ public static void executeSql(
+ String sql,
+ ParameterTool parameter,
+ TableEnvironment context,
+ Consumer<String> callbackFunc) {
+ String flinkSql = resolveSql(sql, parameter);
+ ExecutionContext ctx = new ExecutionContext(context, callbackFunc, parameter);
+ List<SqlCommandCall> calls = SqlCommandParser.parseSQL(flinkSql, null);
+ for (SqlCommandCall call : calls) {
+ processCommand(call, ctx);
+ }
+ finishExecution(flinkSql, ctx);
+ }
+
+ private static String resolveSql(String sql, ParameterTool parameter) {
+ String flinkSql =
+ StringUtils.isBlank(sql)
+ ? parameter.get(ConfigKeys.KEY_FLINK_SQL())
+ : parameter.get(sql);
+ if (StringUtils.isBlank(flinkSql)) {
+ throw new IllegalArgumentException("verify failed: flink sql cannot be empty");
+ }
+ return flinkSql;
+ }
+
+ private static void processCommand(SqlCommandCall call, ExecutionContext ctx) {
+ COMMAND_HANDLERS.getOrDefault(call.command, FlinkSqlExecutor::executeDefault).handle(call, ctx);
+ }
+
+ private static void finishExecution(String flinkSql, ExecutionContext ctx) {
+ if (ctx.hasInsert) {
+ TableResult result = ctx.statementSet.execute();
+ if (result != null) {
+ result.getJobClient()
+ .ifPresent(
+ jobClient -> {
+ try {
+ LOG.info("jobId:{}", jobClient.getJobID());
+ } catch (Exception ignored) {
+ // ignore
+ }
+ });
+ }
+ } else {
+ LOG.error("No 'INSERT' statement to trigger the execution of the Flink job.");
+ throw new IllegalStateException(
+ "No 'INSERT' statement to trigger the execution of the Flink job.");
+ }
+
+ LOG.info(
+ "\n\n\n==============flinkSql==============\n\n {}\n\n============================\n\n\n",
+ flinkSql);
+ }
+
+ private static Map<SqlCommand, CommandHandler> buildCommandHandlers() {
+ Map<SqlCommand, CommandHandler> handlers = new EnumMap<>(SqlCommand.class);
+ handlers.put(SqlCommand.SHOW_CATALOGS,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listCatalogs())));
+ handlers.put(
+ SqlCommand.SHOW_CURRENT_CATALOG,
+ (call, ctx) -> showMeta(call, ctx, ctx.context.getCurrentCatalog()));
+ handlers.put(
+ SqlCommand.SHOW_DATABASES,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listDatabases())));
+ handlers.put(
+ SqlCommand.SHOW_CURRENT_DATABASE,
+ (call, ctx) -> showMeta(call, ctx, ctx.context.getCurrentDatabase()));
+ handlers.put(SqlCommand.SHOW_TABLES, FlinkSqlExecutor::showTables);
+ handlers.put(
+ SqlCommand.SHOW_FUNCTIONS,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listUserDefinedFunctions())));
+ handlers.put(
+ SqlCommand.SHOW_MODULES,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listModules())));
+ handlers.put(SqlCommand.DESC, FlinkSqlExecutor::describeTable);
+ handlers.put(SqlCommand.DESCRIBE, FlinkSqlExecutor::describeTable);
+ handlers.put(SqlCommand.EXPLAIN, FlinkSqlExecutor::explainSql);
+ handlers.put(SqlCommand.SET, FlinkSqlExecutor::setConfig);
+ handlers.put(SqlCommand.RESET, FlinkSqlExecutor::resetConfig);
+ handlers.put(SqlCommand.RESET_ALL, FlinkSqlExecutor::resetConfig);
+ handlers.put(SqlCommand.BEGIN_STATEMENT_SET, FlinkSqlExecutor::warnStatementSet);
+ handlers.put(SqlCommand.END_STATEMENT_SET, FlinkSqlExecutor::warnStatementSet);
+ handlers.put(SqlCommand.INSERT, FlinkSqlExecutor::addInsert);
+ handlers.put(SqlCommand.SELECT, FlinkSqlExecutor::rejectSelect);
+ handlers.put(SqlCommand.DELETE, FlinkSqlExecutor::validateBatchCommand);
+ handlers.put(SqlCommand.UPDATE, FlinkSqlExecutor::validateBatchCommand);
+ return handlers;
+ }
+
+ private static void showMeta(SqlCommandCall call, ExecutionContext ctx, String payload) {
+ ctx.callback.accept(call.command.getName() + ": " + payload);
+ }
+
+ private static void showTables(SqlCommandCall call, ExecutionContext ctx) {
+ String tables =
+ Arrays.stream(ctx.context.listTables())
+ .filter(t -> !t.startsWith("UnnamedTable"))
+ .collect(Collectors.joining("\n"));
+ showMeta(call, ctx, tables);
+ }
+
+ private static void describeTable(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
+ ResolvedSchema schema = ctx.context.from(args).getResolvedSchema();
+ StringBuilder builder = new StringBuilder();
+ builder.append("Column\tType\n");
+ for (int i = 0; i < schema.getColumnCount(); i++) {
+ builder.append(schema.getColumnNames().get(i))
+ .append("\t")
+ .append(schema.getColumnDataTypes().get(i))
+ .append("\n");
+ }
+ ctx.callback.accept(builder.toString());
+ }
+
+ private static void explainSql(SqlCommandCall call, ExecutionContext ctx) {
+ TableResult tableResult = ctx.context.executeSql(call.originSql);
+ Row row = tableResult.collect().next();
+ ctx.callback.accept(row.getField(0).toString());
+ }
+
+ private static void setConfig(SqlCommandCall call, ExecutionContext ctx) {
+ AssertUtils.required(
+ call.operands != null && call.operands.length >= 2,
+ "SET command requires key and value operands");
+ String args = call.operands[0];
+ String operand = call.operands[1];
+ LOG.info("{}: {} --> {}", call.command.getName(), args, operand);
+ ctx.context.getConfig().getConfiguration().setString(args, operand);
+ }
+
+ private static void resetConfig(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
+ try {
+ java.lang.reflect.Field confDataField =
+ Configuration.class.getDeclaredField("confData");
+ confDataField.setAccessible(true);
+ @SuppressWarnings("unchecked")
+ HashMap<String, Object> confData =
+ (HashMap<String, Object>) confDataField.get(ctx.context.getConfig().getConfiguration());
+ synchronized (confData) {
+ if (call.command == SqlCommand.RESET) {
+ confData.remove(args);
+ } else {
+ confData.clear();
+ }
+ }
+ LOG.info("{}: {}", call.command.getName(), args);
+ } catch (ReflectiveOperationException e) {
+ throw new IllegalStateException("Failed to reset Flink table configuration", e);
+ }
+ }
+
+ private static void warnStatementSet(SqlCommandCall call, ExecutionContext ctx) {
+ LOG.warn("SQL Client Syntax: {} ", call.command.getName());
+ }
+
+ private static void addInsert(SqlCommandCall call, ExecutionContext ctx) {
+ ctx.statementSet.addInsertSql(call.originSql);
+ ctx.hasInsert = true;
+ }
+
+ private static void rejectSelect(SqlCommandCall call, ExecutionContext ctx) {
+ LOG.error("StreamPark dose not support 'SELECT' statement now!");
+ throw new UnsupportedOperationException("StreamPark dose not support 'select' statement now!");
+ }
+
+ private static void validateBatchCommand(SqlCommandCall call, ExecutionContext ctx) {
+ String runMode = ctx.parameter.get(ExecutionOptions.RUNTIME_MODE.key());
+ AssertUtils.required(
+ !"STREAMING".equals(runMode),
+ "Currently, "
+ + call.command.getName().toUpperCase()
+ + " statement only supports in batch mode, "
+ + "and it requires the target table connector implements the SupportsRowLevelDelete, "
+ + "For more details please refer to: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/"
+ + call.command.getName());
+ }
+
+ private static void executeDefault(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
+ try {
+ LOCK.lock();
+ ctx.context.executeSql(call.originSql);
+ LOG.info("{}:{}", call.command.getName(), args);
+ } finally {
+ if (LOCK.isHeldByCurrentThread()) {
+ LOCK.unlock();
+ }
+ }
+ }
+
+ private static String firstOperand(SqlCommandCall call) {
+ return call.operands.length == 0 ? null : call.operands[0];
+ }
+
+ private static String joinLines(String[] values) {
+ return String.join("\n", values);
+ }
+
+ @FunctionalInterface
+ private interface CommandHandler {
+
+ void handle(SqlCommandCall call, ExecutionContext ctx);
+ }
+
+ private static final class ExecutionContext {
+
+ private final TableEnvironment context;
+ private final Consumer<String> callback;
+ private final ParameterTool parameter;
+ private final org.apache.flink.table.api.StatementSet statementSet;
+ private boolean hasInsert;
+
+ private ExecutionContext(
+ TableEnvironment context, Consumer<String> callbackFunc,
+ ParameterTool parameter) {
+ this.context = context;
+ this.parameter = parameter;
+ this.statementSet = context.createStatementSet();
+ this.callback =
+ r -> {
+ if (callbackFunc != null) {
+ callbackFunc.accept(r);
+ } else {
+ LOG.info(r);
+ }
+ };
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTraitV2.java
similarity index 59%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTraitV2.java
index d9006d515..63b5c2d39 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTraitV2.java
@@ -24,27 +24,19 @@ import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.cache.DistributedCache;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.io.FileInputFormat;
-import org.apache.flink.api.common.io.FilePathFilter;
import org.apache.flink.api.common.io.InputFormat;
-import org.apache.flink.api.common.restartstrategy.RestartStrategies;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.connector.source.Source;
import org.apache.flink.api.connector.source.SourceSplit;
import org.apache.flink.api.java.tuple.Tuple2;
-import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.ReadableConfig;
import org.apache.flink.core.execution.JobClient;
import org.apache.flink.core.execution.JobListener;
-import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
-import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.CheckpointConfig;
-import org.apache.flink.streaming.api.functions.source.FileMonitoringFunction;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.FileProcessingMode;
-import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.scala.DataStream;
-import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment;
import org.apache.flink.table.api.CompiledPlan;
import org.apache.flink.table.api.ExplainDetail;
import org.apache.flink.table.api.ExplainFormat;
@@ -53,98 +45,96 @@ import org.apache.flink.table.api.Schema;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableDescriptor;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.TableResult;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.Catalog;
import org.apache.flink.table.catalog.CatalogDescriptor;
import org.apache.flink.table.connector.ChangelogMode;
import org.apache.flink.table.expressions.Expression;
-import org.apache.flink.table.functions.AggregateFunction;
import org.apache.flink.table.functions.ScalarFunction;
-import org.apache.flink.table.functions.TableAggregateFunction;
-import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.table.functions.UserDefinedFunction;
import org.apache.flink.table.module.Module;
import org.apache.flink.table.module.ModuleEntry;
import org.apache.flink.table.resource.ResourceUri;
import org.apache.flink.table.types.AbstractDataType;
import org.apache.flink.types.Row;
+import org.apache.flink.util.ParameterTool;
import org.apache.flink.util.SplittableIterator;
-import com.esotericsoftware.kryo.Serializer;
-
+import java.util.Collection;
import java.util.List;
import java.util.Optional;
-
-import scala.Function1;
-import scala.Serializable;
-import scala.collection.Iterator;
-import scala.collection.Seq;
-import scala.runtime.BoxedUnit;
+import java.util.function.Consumer;
/**
- * Integration api of stream and table.
+ * Integration API of stream and table environments for Flink 2.x.
*
- * <p>Methods prefixed with {@code $} implement the Flink Scala {@code StreamTableEnvironment} API
- * and retain Scala-compatible names required by that interface.
+ * <p>Once a Table has been converted to a DataStream, the DataStream job must be executed using the
+ * execute method of the StreamExecutionEnvironment.
*/
-public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
+public abstract class FlinkStreamTableTraitV2 implements StreamTableEnvironment {
public final ParameterTool parameter;
- public boolean isConvertedToDataStream = false;
private final StreamExecutionEnvironment streamEnv;
+
private final StreamTableEnvironment tableEnv;
- protected FlinkStreamTableTrait(
- ParameterTool parameter,
- StreamExecutionEnvironment streamEnv,
- StreamTableEnvironment tableEnv) {
+ /** Whether a table has been converted to a DataStream. */
+ public boolean isConvertedToDataStream;
+
+ protected FlinkStreamTableTraitV2(
+ ParameterTool parameter,
+ StreamExecutionEnvironment streamEnv,
+ StreamTableEnvironment tableEnv) {
this.parameter = parameter;
this.streamEnv = streamEnv;
this.tableEnv = tableEnv;
}
- protected StreamExecutionEnvironment streamEnv() {
+ protected StreamExecutionEnvironment getStreamEnv() {
return streamEnv;
}
- protected StreamTableEnvironment tableEnv() {
+ protected StreamTableEnvironment getStreamTableEnv() {
return tableEnv;
}
- /**
- * Once a Table has been converted to a DataStream, the DataStream job must be executed using the
- * execute method of the StreamExecutionEnvironment.
- */
+ /** Recommended API to start tasks. */
public JobExecutionResult start() {
return start(null);
}
- /** Recommended to use this Api to start tasks */
public JobExecutionResult start(String name) {
- String appName = FlinkEnvironmentUtils.getAppName(parameter, name, true);
+ String appName = FlinkParameterUtils.getAppName(parameter, name, true);
return execute(appName);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
public JobExecutionResult execute(String jobName) {
Utils.printLogo("FlinkStreamTable " + jobName + " Starting...");
if (isConvertedToDataStream) {
- return streamEnv.execute(jobName);
+ try {
+ return streamEnv.execute(jobName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
}
return null;
}
- public void sql() {
- sql(null);
+ public void sql(String sql) {
+ sql(sql, null);
}
- public void sql(String sql) {
- FlinkSqlExecutor.executeSql(sql, parameter, this);
+ public void sql(String sql, Consumer<String> callback) {
+ FlinkSqlExecutor.executeSql(sql, parameter, this, callback);
}
- public org.apache.flink.streaming.api.environment.StreamExecutionEnvironment getJavaEnv() {
- return streamEnv.getJavaEnv();
+ public StreamExecutionEnvironment getJavaEnv() {
+ return streamEnv;
}
public List<Tuple2<String, DistributedCache.DistributedCacheEntry>> $getCachedFiles() {
@@ -203,135 +193,52 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.getCheckpointingMode();
}
- public StreamExecutionEnvironment $setStateBackend(StateBackend backend) {
- return streamEnv.setStateBackend(backend);
- }
-
- public StateBackend $getStateBackend() {
- return streamEnv.getStateBackend();
- }
-
- public void $setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) {
- streamEnv.setRestartStrategy(restartStrategyConfiguration);
- }
-
- public RestartStrategies.RestartStrategyConfiguration $getRestartStrategy() {
- return streamEnv.getRestartStrategy();
- }
-
- public void $setNumberOfExecutionRetries(int numRetries) {
- streamEnv.setNumberOfExecutionRetries(numRetries);
- }
-
- public int $getNumberOfExecutionRetries() {
- return streamEnv.getNumberOfExecutionRetries();
- }
-
- public <T extends Serializer<?> & Serializable> void $addDefaultKryoSerializer(
- Class<?> type,
- T serializer) {
- streamEnv.addDefaultKryoSerializer(type, serializer);
- }
-
- public void $addDefaultKryoSerializer(Class<?> type, Class<? extends Serializer<?>> serializerClass) {
- streamEnv.addDefaultKryoSerializer(type, serializerClass);
- }
-
- public <T extends Serializer<?> & Serializable> void $registerTypeWithKryoSerializer(
- Class<?> clazz,
- T serializer) {
- streamEnv.registerTypeWithKryoSerializer(clazz, serializer);
- }
-
- public void $registerTypeWithKryoSerializer(
- Class<?> clazz,
- Class<? extends Serializer<?>> serializer) {
- streamEnv.registerTypeWithKryoSerializer(clazz, serializer);
- }
-
- public void $registerType(Class<?> typeClass) {
- streamEnv.registerType(typeClass);
- }
-
- public TimeCharacteristic $getStreamTimeCharacteristic() {
- return streamEnv.getStreamTimeCharacteristic();
- }
-
public void $configure(ReadableConfig configuration, ClassLoader classLoader) {
streamEnv.configure(configuration, classLoader);
}
- public DataStream<Object> $fromSequence(long from, long to) {
- return streamEnv.fromSequence(from, to);
- }
-
- public <T> DataStream<T> $fromElements(Seq<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromElements(data, typeInfo);
- }
-
- public <T> DataStream<T> $fromCollection(Seq<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromCollection(data, typeInfo);
- }
-
- public <T> DataStream<T> $fromCollection(Iterator<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromCollection(data, typeInfo);
+ public <T> DataStream<T> $fromData(T data) {
+ return streamEnv.fromData(data);
}
- public <T> DataStream<T> $fromParallelCollection(
- SplittableIterator<T> data,
- TypeInformation<T> typeInfo) {
- return streamEnv.fromParallelCollection(data, typeInfo);
+ public DataStream<Long> $fromSequence(long from, long to) {
+ return streamEnv.fromSequence(from, to);
}
- public DataStream<String> $readTextFile(String filePath) {
- return streamEnv.readTextFile(filePath);
+ public <T> DataStream<T> $fromCollection(Collection<T> data) {
+ return streamEnv.fromCollection(data);
}
- public DataStream<String> $readTextFile(String filePath, String charsetName) {
- return streamEnv.readTextFile(filePath, charsetName);
+ public <T> DataStream<T> $fromParallelCollection(SplittableIterator<T> data, Class<T> clazz) {
+ return streamEnv.fromParallelCollection(data, clazz);
}
- public <T> DataStream<T> $readFile(FileInputFormat<T> inputFormat, String filePath, TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, typeInfo);
+ public <T> DataStream<T> $readFile(FileInputFormat<T> inputFormat, String filePath) {
+ return streamEnv.readFile(inputFormat, filePath);
}
public <T> DataStream<T> $readFile(
FileInputFormat<T> inputFormat,
String filePath,
FileProcessingMode watchType,
- long interval,
- TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, watchType, interval, typeInfo);
+ long interval) {
+ return streamEnv.readFile(inputFormat, filePath, watchType, interval);
}
public DataStream<String> $socketTextStream(
- String hostname,
- int port,
- char delimiter,
- long maxRetry) {
+ String hostname, int port, char delimiter, long maxRetry) {
return streamEnv.socketTextStream(hostname, port, delimiter, maxRetry);
}
- public <T> DataStream<T> $createInput(InputFormat<T, ?> inputFormat, TypeInformation<T> typeInfo) {
- return streamEnv.createInput(inputFormat, typeInfo);
- }
-
- public <T> DataStream<T> $addSource(SourceFunction<T> function, TypeInformation<T> typeInfo) {
- return streamEnv.addSource(function, typeInfo);
- }
-
- public <T> DataStream<T> $addSource(
- Function1<SourceFunction.SourceContext<T>, BoxedUnit> function,
- TypeInformation<T> typeInfo) {
- return streamEnv.addSource(function, typeInfo);
+ public <T> DataStream<T> $createInput(InputFormat<T, ?> inputFormat) {
+ return streamEnv.createInput(inputFormat);
}
public <T> DataStream<T> $fromSource(
Source<T, ? extends SourceSplit, ?> source,
WatermarkStrategy<T> watermarkStrategy,
- String sourceName,
- TypeInformation<T> typeInfo) {
- return streamEnv.fromSource(source, watermarkStrategy, sourceName, typeInfo);
+ String sourceName) {
+ return streamEnv.fromSource(source, watermarkStrategy, sourceName);
}
public void $registerJobListener(JobListener jobListener) {
@@ -343,11 +250,19 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
public JobClient $executeAsync() {
- return streamEnv.executeAsync();
+ try {
+ return streamEnv.executeAsync();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
}
public JobClient $executeAsync(String jobName) {
- return streamEnv.executeAsync(jobName);
+ try {
+ return streamEnv.executeAsync(jobName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
}
public String $getExecutionPlan() {
@@ -358,8 +273,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.getStreamGraph();
}
- public org.apache.flink.streaming.api.environment.StreamExecutionEnvironment $getWrappedStreamExecutionEnvironment() {
- return streamEnv.getWrappedStreamExecutionEnvironment();
+ public StreamExecutionEnvironment $getWrappedStreamExecutionEnvironment() {
+ return streamEnv;
}
public void $registerCachedFile(String filePath, String name) {
@@ -378,48 +293,6 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.isForceUnalignedCheckpoints();
}
- public StreamExecutionEnvironment $enableCheckpointing(
- long interval,
- CheckpointingMode mode,
- boolean force) {
- return streamEnv.enableCheckpointing(interval, mode, force);
- }
-
- public StreamExecutionEnvironment $enableCheckpointing() {
- return streamEnv.enableCheckpointing();
- }
-
- public DataStream<Object> $generateSequence(long from, long to) {
- return streamEnv.generateSequence(from, to);
- }
-
- public DataStream<String> $readFileStream(
- String streamPath,
- long intervalMillis,
- FileMonitoringFunction.WatchType watchType) {
- return streamEnv.readFileStream(streamPath, intervalMillis, watchType);
- }
-
- public <T> DataStream<T> $readFile(
- FileInputFormat<T> inputFormat,
- String filePath,
- FileProcessingMode watchType,
- long interval,
- FilePathFilter filter,
- TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, watchType, interval, filter, typeInfo);
- }
-
- @Override
- public <T> Table fromDataStream(DataStream<T> dataStream) {
- return tableEnv.fromDataStream(dataStream);
- }
-
- @Override
- public <T> Table fromDataStream(DataStream<T> dataStream, Seq<Expression> fields) {
- return tableEnv.fromDataStream(dataStream, fields);
- }
-
@Override
public <T> Table fromDataStream(DataStream<T> dataStream, Schema schema) {
return tableEnv.fromDataStream(dataStream, schema);
@@ -436,20 +309,11 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public Table fromChangelogStream(DataStream<Row> dataStream, Schema schema, ChangelogMode changelogMode) {
+ public Table fromChangelogStream(
+ DataStream<Row> dataStream, Schema schema, ChangelogMode changelogMode) {
return tableEnv.fromChangelogStream(dataStream, schema, changelogMode);
}
- @Override
- public <T> void createTemporaryView(String path, DataStream<T> dataStream) {
- tableEnv.createTemporaryView(path, dataStream);
- }
-
- @Override
- public <T> void createTemporaryView(String path, DataStream<T> dataStream, Seq<Expression> fields) {
- tableEnv.createTemporaryView(path, dataStream, fields);
- }
-
@Override
public <T> void createTemporaryView(String path, DataStream<T> dataStream, Schema schema) {
tableEnv.createTemporaryView(path, dataStream, schema);
@@ -486,21 +350,121 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public DataStream<Row> toChangelogStream(Table table, Schema targetSchema, ChangelogMode changelogMode) {
+ public DataStream<Row> toChangelogStream(
+ Table table, Schema targetSchema, ChangelogMode changelogMode) {
isConvertedToDataStream = true;
return tableEnv.toChangelogStream(table, targetSchema, changelogMode);
}
@Override
- public <T> DataStream<T> toAppendStream(Table table, TypeInformation<T> typeInfo) {
+ public <T> DataStream<T> toAppendStream(Table table, Class<T> clazz) {
isConvertedToDataStream = true;
- return tableEnv.toAppendStream(table, typeInfo);
+ return tableEnv.toAppendStream(table, clazz);
}
@Override
- public <T> DataStream<scala.Tuple2<Object, T>> toRetractStream(Table table, TypeInformation<T> typeInfo) {
+ public <T> DataStream<Tuple2<Boolean, T>> toRetractStream(Table table, Class<T> clazz) {
isConvertedToDataStream = true;
- return tableEnv.toRetractStream(table, typeInfo);
+ return tableEnv.toRetractStream(table, clazz);
+ }
+
+ @Override
+ public void createCatalog(String catalogName, CatalogDescriptor catalogDescriptor) {
+ tableEnv.createCatalog(catalogName, catalogDescriptor);
+ }
+
+ @Override
+ public void useModules(String... moduleNames) {
+ tableEnv.useModules(moduleNames);
+ }
+
+ @Override
+ public void createFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createFunction(path, className, resourceUris);
+ }
+
+ @Override
+ public void createFunction(
+ String path,
+ String className,
+ List<ResourceUri> resourceUris,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
+ }
+
+ @Override
+ public void createTemporaryFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporaryFunction(path, className, resourceUris);
+ }
+
+ @Override
+ public void createTemporarySystemFunction(
+ String name, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporarySystemFunction(name, className, resourceUris);
+ }
+
+ @Override
+ public void createTemporaryTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTemporaryTable(path, descriptor);
+ }
+
+ @Override
+ public void createTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTable(path, descriptor);
+ }
+
+ @Override
+ public Table from(TableDescriptor descriptor) {
+ return tableEnv.from(descriptor);
+ }
+
+ @Override
+ public ModuleEntry[] listFullModules() {
+ return tableEnv.listFullModules();
+ }
+
+ @Override
+ public String[] listTables(String catalogName, String databaseName) {
+ return tableEnv.listTables(catalogName, databaseName);
+ }
+
+ @Override
+ public String explainSql(
+ String statement, ExplainFormat format, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, format, extraDetails);
+ }
+
+ @Override
+ public CompiledPlan loadPlan(PlanReference planReference) throws TableException {
+ return tableEnv.loadPlan(planReference);
+ }
+
+ @Override
+ public CompiledPlan compilePlanSql(String statement) throws TableException {
+ return tableEnv.compilePlanSql(statement);
+ }
+
+ @Override
+ public <T> Table fromDataStream(DataStream<T> dataStream) {
+ return tableEnv.fromDataStream(dataStream);
+ }
+
+ @Override
+ public <T> Table fromDataStream(DataStream<T> dataStream, Expression... fields) {
+ return tableEnv.fromDataStream(dataStream, fields);
+ }
+
+ @Override
+ public <T> void createTemporaryView(String path, DataStream<T> dataStream) {
+ tableEnv.createTemporaryView(path, dataStream);
+ }
+
+ @Override
+ public <T> void createTemporaryView(
+ String path, DataStream<T> dataStream, Expression... fields) {
+ tableEnv.createTemporaryView(path, dataStream, fields);
}
@Override
@@ -545,13 +509,13 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
@Override
public void createTemporarySystemFunction(
- String name,
- Class<? extends UserDefinedFunction> functionClass) {
+ String name, Class<? extends UserDefinedFunction> functionClass) {
tableEnv.createTemporarySystemFunction(name, functionClass);
}
@Override
- public void createTemporarySystemFunction(String name, UserDefinedFunction functionInstance) {
+ public void createTemporarySystemFunction(
+ String name, UserDefinedFunction functionInstance) {
tableEnv.createTemporarySystemFunction(name, functionInstance);
}
@@ -579,7 +543,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public void createTemporaryFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
+ public void createTemporaryFunction(
+ String path, Class<? extends UserDefinedFunction> functionClass) {
tableEnv.createTemporaryFunction(path, functionClass);
}
@@ -698,127 +663,29 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return tableEnv.getConfig();
}
- @Override
- public <T> void registerFunction(String name, TableFunction<T> tf, TypeInformation<T> typeInfo) {
- tableEnv.registerFunction(name, tf, typeInfo);
- }
-
- @Override
- public <T, A> void registerFunction(
- String name,
- AggregateFunction<T, A> f,
- TypeInformation<T> typeInfo1,
- TypeInformation<A> typeInfo2) {
- tableEnv.registerFunction(name, f, typeInfo1, typeInfo2);
- }
-
- @Override
- public <T, A> void registerFunction(
- String name,
- TableAggregateFunction<T, A> f,
- TypeInformation<T> typeInfo1,
- TypeInformation<A> typeInfo2) {
- tableEnv.registerFunction(name, f, typeInfo1, typeInfo2);
- }
-
- @Override
- public <T> void registerDataStream(String name, DataStream<T> dataStream) {
- tableEnv.registerDataStream(name, dataStream);
- }
-
- @Override
- public <T> void registerDataStream(String name, DataStream<T> dataStream, Seq<Expression> fields) {
- tableEnv.registerDataStream(name, dataStream, fields);
- }
-
- @Override
- public void registerFunction(String name, ScalarFunction function) {
- tableEnv.registerFunction(name, function);
- }
-
- @Override
- public void registerTable(String name, Table table) {
- tableEnv.registerTable(name, table);
- }
-
- @Override
- public Table scan(String... tablePath) {
- return tableEnv.scan(tablePath);
- }
-
@Override
public String[] getCompletionHints(String statement, int position) {
return tableEnv.getCompletionHints(statement, position);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void useModules(String... modules) {
- tableEnv.useModules(modules);
- }
-
- @Override
- public ModuleEntry[] listFullModules() {
- return tableEnv.listFullModules();
- }
-
- @Override
- public void createTemporaryTable(String path, TableDescriptor descriptor) {
- tableEnv.createTemporaryTable(path, descriptor);
- }
-
- @Override
- public void createTable(String path, TableDescriptor descriptor) {
- tableEnv.createTable(path, descriptor);
- }
-
- @Override
- public Table from(TableDescriptor descriptor) {
- return tableEnv.from(descriptor);
- }
-
- @Override
- public String[] listTables(String catalogName, String databaseName) {
- return tableEnv.listTables(catalogName, databaseName);
- }
-
- @Override
- public CompiledPlan loadPlan(PlanReference planReference) {
- return tableEnv.loadPlan(planReference);
- }
-
- @Override
- public CompiledPlan compilePlanSql(String stmt) {
- return tableEnv.compilePlanSql(stmt);
- }
-
- @Override
- public void createFunction(String path, String className, List<ResourceUri> resourceUris) {
- tableEnv.createFunction(path, className, resourceUris);
- }
-
- @Override
- public void createFunction(
- String path, String className, List<ResourceUri> resourceUris, boolean ignoreIfExists) {
- tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
- }
-
- @Override
- public void createTemporaryFunction(String path, String className, List<ResourceUri> resourceUris) {
- tableEnv.createTemporaryFunction(path, className, resourceUris);
- }
-
- @Override
- public void createTemporarySystemFunction(String name, String className, List<ResourceUri> resourceUris) {
- tableEnv.createTemporarySystemFunction(name, className, resourceUris);
+ public Table scan(String... tablePath) {
+ return tableEnv.scan(tablePath);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public String explainSql(String statement, ExplainFormat format, ExplainDetail... extraDetails) {
- return tableEnv.explainSql(statement, format, extraDetails);
+ public void registerTable(String name, Table table) {
+ tableEnv.registerTable(name, table);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createCatalog(String catalog, CatalogDescriptor catalogDescriptor) {
- tableEnv.createCatalog(catalog, catalogDescriptor);
+ public void registerFunction(String name, ScalarFunction function) {
+ tableEnv.registerFunction(name, function);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializerV2.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializerV2.java
new file mode 100644
index 000000000..f412ce4af
--- /dev/null
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializerV2.java
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.core;
+
+import org.apache.streampark.common.conf.ConfigKeys;
+import org.apache.streampark.common.util.DeflaterUtils;
+import org.apache.streampark.common.util.FileUtils;
+import org.apache.streampark.common.util.HdfsUtils;
+import org.apache.streampark.common.util.PropertiesUtils;
+import org.apache.streampark.flink.core.conf.FlinkConfiguration;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.util.ParameterTool;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Initializes Flink streaming execution environment from application arguments. */
+public class FlinkStreamingInitializerV2 {
+
+ final String[] args;
+
+ StreamEnvConfigFunction javaStreamEnvConfFunc;
+
+ private FlinkConfiguration configuration;
+
+ private StreamExecutionEnvironment streamEnv;
+
+ FlinkStreamingInitializerV2(String[] args) {
+ this.args = args;
+ }
+
+ public static StreamingInitResult initialize(String[] args, StreamEnvConfigFunction config) {
+ FlinkStreamingInitializerV2 flinkInitializer = new FlinkStreamingInitializerV2(args);
+ flinkInitializer.javaStreamEnvConfFunc = config;
+ return new StreamingInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getStreamEnv());
+ }
+
+ public static StreamingInitResult initialize(StreamEnvConfig args) {
+ FlinkStreamingInitializerV2 flinkInitializer = new FlinkStreamingInitializerV2(args.args);
+ flinkInitializer.javaStreamEnvConfFunc = args.conf;
+ return new StreamingInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getStreamEnv());
+ }
+
+ ParameterTool getParameter() {
+ return getConfiguration().parameter;
+ }
+
+ FlinkConfiguration getConfiguration() {
+ if (configuration == null) {
+ configuration = initParameter();
+ }
+ return configuration;
+ }
+
+ StreamExecutionEnvironment getStreamEnv() {
+ if (streamEnv == null) {
+ streamEnv =
+ StreamExecutionEnvironment.getExecutionEnvironment(
+ getConfiguration().envConfig);
+ if (javaStreamEnvConfFunc != null) {
+ javaStreamEnvConfFunc.configuration(streamEnv, getConfiguration().parameter);
+ }
+ streamEnv.getConfig().setGlobalJobParameters(getConfiguration().parameter);
+ }
+ return streamEnv;
+ }
+
+ FlinkConfiguration initParameter() {
+ ParameterTool argsMap = ParameterTool.fromArgs(args);
+ String configFile = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
+ if (configFile == null || configFile.isEmpty()) {
+ throw new ExceptionInInitializerError(
+ "[StreamPark] Usage:can't find config,please set \"--conf $path \" in main arguments");
+ }
+ Map<String, String> configMap = parseConfig(configFile);
+ Map<String, String> properConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_PROPERTY_PREFIX());
+ Map<String, String> appConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
+
+ ParameterTool parameter =
+ ParameterTool.fromSystemProperties()
+ .mergeWith(ParameterTool.fromMap(properConf))
+ .mergeWith(ParameterTool.fromMap(appConf))
+ .mergeWith(argsMap);
+
+ Configuration envConfig = Configuration.fromMap(properConf);
+ return new FlinkConfiguration(parameter, envConfig, null);
+ }
+
+ Map<String, String> parseConfig(String config) {
+ Map<String, String> map;
+ if (config.startsWith("yaml://")) {
+ map = PropertiesUtils.fromYamlText(DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("conf://")) {
+ map = PropertiesUtils.fromHoconText(DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("prop://")) {
+ map =
+ PropertiesUtils.fromPropertiesText(
+ DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("hdfs://")) {
+ try {
+ String text = HdfsUtils.read(config);
+ map = readConfig(config, text);
+ } catch (IOException e) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Failed to read application config from HDFS: " + config, e);
+ }
+ } else {
+ File file = new File(config);
+ if (!file.exists()) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Usage: application config file: "
+ + file
+ + " is not found!!!");
+ }
+ try {
+ map = readConfig(config, FileUtils.readFile(file));
+ } catch (IOException e) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Failed to read application config file: " + config, e);
+ }
+ }
+ Map<String, String> filtered = new HashMap<>();
+ map.forEach(
+ (key, value) -> {
+ if (value != null && !value.isEmpty()) {
+ filtered.put(key, value);
+ }
+ });
+ return filtered;
+ }
+
+ private Map<String, String> readConfig(String config, String text) {
+ String format = config.substring(config.lastIndexOf('.') + 1).toLowerCase();
+ switch (format) {
+ case "yml":
+ case "yaml":
+ return PropertiesUtils.fromYamlText(text);
+ case "conf":
+ return PropertiesUtils.fromHoconText(text);
+ case "properties":
+ return PropertiesUtils.fromPropertiesText(text);
+ default:
+ throw new IllegalArgumentException(
+ "[StreamPark] Usage: application config file error,must be [yaml|conf|properties]");
+ }
+ }
+
+ Map<String, String> extractConfigByPrefix(Map<String, String> configMap, String prefix) {
+ Map<String, String> map = new HashMap<>();
+ configMap.forEach(
+ (key, value) -> {
+ if (key.startsWith(prefix)) {
+ map.put(key.substring(prefix.length()), value);
+ }
+ });
+ return map;
+ }
+
+ /** Streaming initialization result. */
+ public static final class StreamingInitResult {
+
+ public final ParameterTool parameter;
+ public final StreamExecutionEnvironment streamEnv;
+
+ StreamingInitResult(ParameterTool parameter, StreamExecutionEnvironment streamEnv) {
+ this.parameter = parameter;
+ this.streamEnv = streamEnv;
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializerV2.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializerV2.java
new file mode 100644
index 000000000..7799c1e2b
--- /dev/null
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializerV2.java
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.core;
+
+import org.apache.streampark.common.conf.ConfigKeys;
+import org.apache.streampark.common.enums.PlannerType;
+import org.apache.streampark.common.util.DeflaterUtils;
+import org.apache.streampark.common.util.PropertiesUtils;
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
+import org.apache.streampark.flink.core.conf.FlinkConfiguration;
+
+import org.apache.streampark.shaded.org.slf4j.Logger;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.util.ParameterTool;
+
+import java.io.File;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Initializes Flink table and stream-table environments from application arguments. */
+public class FlinkTableInitializerV2 extends FlinkStreamingInitializerV2 {
+
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory()
+ .getLogger(FlinkTableInitializerV2.class.getName());
+
+ private TableEnvConfigFunction javaTableEnvConfFunc;
+
+ private EnvironmentSettings.Builder envSettingsBuilder;
+
+ private TableEnvironment tableEnv;
+
+ private StreamTableEnvironment streamTableEnv;
+
+ FlinkTableInitializerV2(String[] args) {
+ super(args);
+ }
+
+ public static TableInitResult initialize(TableEnvConfig args) {
+ FlinkTableInitializerV2 flinkInitializer = new FlinkTableInitializerV2(args.args);
+ flinkInitializer.javaTableEnvConfFunc = args.conf;
+ return new TableInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getTableEnv());
+ }
+
+ public static StreamTableInitResult initialize(StreamTableEnvConfig args) {
+ FlinkTableInitializerV2 flinkInitializer = new FlinkTableInitializerV2(args.args);
+ flinkInitializer.javaStreamEnvConfFunc = args.streamConfig;
+ flinkInitializer.javaTableEnvConfFunc = args.tableConfig;
+ return new StreamTableInitResult(
+ flinkInitializer.getConfiguration().parameter,
+ flinkInitializer.getStreamEnv(),
+ flinkInitializer.getStreamTableEnv());
+ }
+
+ public static StreamTableInitResult initialize(
+ String[] args,
+ StreamEnvConfigFunction streamConfig,
+ TableEnvConfigFunction tableConfig) {
+ FlinkTableInitializerV2 flinkInitializer = new FlinkTableInitializerV2(args);
+ flinkInitializer.javaStreamEnvConfFunc = streamConfig;
+ flinkInitializer.javaTableEnvConfFunc = tableConfig;
+ return new StreamTableInitResult(
+ flinkInitializer.getConfiguration().parameter,
+ flinkInitializer.getStreamEnv(),
+ flinkInitializer.getStreamTableEnv());
+ }
+
+ TableEnvironment getTableEnv() {
+ if (tableEnv == null) {
+ LOG.info("job working in batch mode");
+ EnvironmentSettings.Builder builder = getEnvSettingsBuilder();
+ builder.inBatchMode();
+ tableEnv =
+ FlinkParameterUtils.setAppName(
+ TableEnvironment.create(builder.build()), getParameter());
+ applyTableEnvConfig(tableEnv.getConfig());
+ }
+ return tableEnv;
+ }
+
+ StreamTableEnvironment getStreamTableEnv() {
+ if (streamTableEnv == null) {
+ LOG.info("components should work in streaming mode");
+ EnvironmentSettings.Builder builder = getEnvSettingsBuilder();
+ builder.inStreamingMode();
+ EnvironmentSettings setting = builder.build();
+
+ if (javaStreamEnvConfFunc != null) {
+ javaStreamEnvConfFunc.configuration(getStreamEnv(), getParameter());
+ }
+ streamTableEnv =
+ FlinkParameterUtils.setAppName(
+ StreamTableEnvironment.create(getStreamEnv(), setting), getParameter());
+ applyTableEnvConfig(streamTableEnv.getConfig());
+ }
+ return streamTableEnv;
+ }
+
+ private void applyTableEnvConfig(TableConfig config) {
+ if (javaTableEnvConfFunc != null) {
+ javaTableEnvConfFunc.configuration(config, getParameter());
+ }
+ }
+
+ private EnvironmentSettings.Builder getEnvSettingsBuilder() {
+ if (envSettingsBuilder == null) {
+ envSettingsBuilder = buildEnvSettings(getParameter());
+ }
+ return envSettingsBuilder;
+ }
+
+ private EnvironmentSettings.Builder buildEnvSettings(ParameterTool parameter) {
+ EnvironmentSettings.Builder builder = EnvironmentSettings.newInstance();
+
+ PlannerType plannerType = PlannerType.BLINK;
+ String plannerName = parameter.get(ConfigKeys.KEY_FLINK_TABLE_PLANNER(), null);
+ if (plannerName != null && !plannerName.isEmpty()) {
+ try {
+ plannerType = PlannerType.withName(plannerName);
+ } catch (IllegalArgumentException e) {
+ plannerType = PlannerType.BLINK;
+ }
+ }
+
+ switch (plannerType) {
+ case BLINK:
+ invokePlannerMethod(builder, "useBlinkPlanner", "blinkPlanner will be used.");
+ break;
+ case OLD:
+ invokePlannerMethod(builder, "useOldPlanner", "useOldPlanner will be used.");
+ break;
+ case ANY:
+ invokePlannerMethod(builder, "useAnyPlanner", "useAnyPlanner will be used.");
+ break;
+ default:
+ break;
+ }
+
+ String flinkConf = parameter.get(ConfigKeys.KEY_FLINK_CONF(), null);
+ if (flinkConf == null || flinkConf.isEmpty()) {
+ throw new ExceptionInInitializerError(
+ "[StreamPark] Usage:can't find config,please set \"--flink.conf $conf \" in main arguments");
+ }
+ builder.withConfiguration(
+ Configuration.fromMap(
+ PropertiesUtils.fromYamlText(DeflaterUtils.unzipString(flinkConf))));
+
+ String catalog = parameter.get(ConfigKeys.KEY_FLINK_TABLE_CATALOG(), null);
+ String database = parameter.get(ConfigKeys.KEY_FLINK_TABLE_DATABASE(), null);
+ if (catalog != null && database != null) {
+ LOG.info("with built in catalog: {}", catalog);
+ LOG.info("with built in database: {}", database);
+ builder.withBuiltInCatalogName(catalog);
+ builder.withBuiltInDatabaseName(database);
+ } else if (catalog != null) {
+ LOG.info("with built in catalog: {}", catalog);
+ builder.withBuiltInCatalogName(catalog);
+ } else if (database != null) {
+ LOG.info("with built in database: {}", database);
+ builder.withBuiltInDatabaseName(database);
+ }
+ return builder;
+ }
+
+ private void invokePlannerMethod(
+ EnvironmentSettings.Builder builder, String methodName, String successMessage) {
+ try {
+ Method method = builder.getClass().getDeclaredMethod(methodName);
+ method.setAccessible(true);
+ method.invoke(builder);
+ if (successMessage != null) {
+ LOG.info(successMessage);
+ }
+ } catch (NoSuchMethodException e) {
+ LOG.warn("{} deprecated", methodName);
+ } catch (ReflectiveOperationException e) {
+ LOG.warn("Failed to invoke {} on EnvironmentSettings.Builder", methodName, e);
+ }
+ }
+
+ @Override
+ FlinkConfiguration initParameter() {
+ ParameterTool argsMap = ParameterTool.fromArgs(args);
+ String configFile = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
+ FlinkConfiguration configuration;
+ if (configFile == null || configFile.isEmpty()) {
+ LOG.warn("Usage:can't find config,you can set \"--conf $path \" in main arguments");
+ ParameterTool parameter = ParameterTool.fromSystemProperties().mergeWith(argsMap);
+ configuration =
+ new FlinkConfiguration(parameter, new Configuration(), new Configuration());
+ } else {
+ Map<String, String> configMap = parseConfig(configFile);
+ Map<String, String> sqlConf = new HashMap<>();
+ configMap.forEach(
+ (key, value) -> {
+ if (key.startsWith(ConfigKeys.KEY_SQL_PREFIX())) {
+ sqlConf.put(key.substring(ConfigKeys.KEY_SQL_PREFIX().length()), value);
+ }
+ });
+
+ Map<String, String> properConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_PROPERTY_PREFIX());
+ Map<String, String> appConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
+ Map<String, String> tableConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_TABLE_PREFIX());
+
+ Configuration tableConfig = Configuration.fromMap(tableConf);
+ Configuration envConfig = Configuration.fromMap(properConf);
+
+ ParameterTool parameter =
+ ParameterTool.fromSystemProperties()
+ .mergeWith(ParameterTool.fromMap(properConf))
+ .mergeWith(ParameterTool.fromMap(tableConf))
+ .mergeWith(ParameterTool.fromMap(appConf))
+ .mergeWith(ParameterTool.fromMap(sqlConf))
+ .mergeWith(argsMap);
+
+ configuration = new FlinkConfiguration(parameter, envConfig, tableConfig);
+ }
+
+ String flinkSql = configuration.parameter.get(ConfigKeys.KEY_FLINK_SQL(), null);
+ if (flinkSql == null) {
+ return configuration;
+ }
+
+ try {
+ String value = DeflaterUtils.unzipString(flinkSql);
+ return configuration.withParameter(
+ configuration.parameter.mergeWith(
+ ParameterTool.fromMap(
+ Map.of(ConfigKeys.KEY_FLINK_SQL(), value))));
+ } catch (Exception ignored) {
+ File sqlFile = new File(flinkSql);
+ try {
+ Map<String, String> value =
+ PropertiesUtils.fromYamlFile(sqlFile.getAbsolutePath());
+ return configuration.withParameter(
+ configuration.parameter.mergeWith(ParameterTool.fromMap(value)));
+ } catch (Exception e) {
+ throw new IllegalArgumentException("[StreamPark] init sql error." + e, e);
+ }
+ }
+ }
+
+ /** Table initialization result. */
+ public static final class TableInitResult {
+
+ public final ParameterTool parameter;
+ public final TableEnvironment tableEnv;
+
+ public TableInitResult(ParameterTool parameter, TableEnvironment tableEnv) {
+ this.parameter = parameter;
+ this.tableEnv = tableEnv;
+ }
+ }
+
+ /** Stream-table initialization result. */
+ public static final class StreamTableInitResult {
+
+ public final ParameterTool parameter;
+ public final StreamExecutionEnvironment streamEnv;
+ public final StreamTableEnvironment streamTableEnv;
+
+ public StreamTableInitResult(
+ ParameterTool parameter,
+ StreamExecutionEnvironment streamEnv,
+ StreamTableEnvironment streamTableEnv) {
+ this.parameter = parameter;
+ this.streamEnv = streamEnv;
+ this.streamTableEnv = streamTableEnv;
+ }
+ }
+}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
similarity index 58%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
index c8d47444b..e655b3734 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
@@ -20,7 +20,6 @@ package org.apache.streampark.flink.core;
import org.apache.streampark.common.util.Utils;
import org.apache.flink.api.common.JobExecutionResult;
-import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.table.api.CompiledPlan;
import org.apache.flink.table.api.ExplainDetail;
import org.apache.flink.table.api.ExplainFormat;
@@ -30,6 +29,7 @@ import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableDescriptor;
import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.TableResult;
import org.apache.flink.table.catalog.Catalog;
import org.apache.flink.table.catalog.CatalogDescriptor;
@@ -40,13 +40,16 @@ import org.apache.flink.table.module.Module;
import org.apache.flink.table.module.ModuleEntry;
import org.apache.flink.table.resource.ResourceUri;
import org.apache.flink.table.types.AbstractDataType;
+import org.apache.flink.util.ParameterTool;
import java.util.List;
import java.util.Optional;
+/** Base table environment trait with SQL execution helpers. */
public abstract class FlinkTableTrait implements TableEnvironment {
public final ParameterTool parameter;
+
private final TableEnvironment tableEnv;
protected FlinkTableTrait(ParameterTool parameter, TableEnvironment tableEnv) {
@@ -54,315 +57,326 @@ public abstract class FlinkTableTrait implements TableEnvironment {
this.tableEnv = tableEnv;
}
- protected TableEnvironment delegate() {
+ protected TableEnvironment getTableEnv() {
return tableEnv;
}
public JobExecutionResult start() {
- String appName = FlinkEnvironmentUtils.getAppName(parameter, null, true);
- return printStartupLogo(appName);
+ String appName = FlinkParameterUtils.getAppName(parameter, true);
+ return execute(appName);
}
- JobExecutionResult printStartupLogo(String jobName) {
+ public JobExecutionResult execute(String jobName) {
Utils.printLogo("FlinkTable " + jobName + " Starting...");
return null;
}
- public void sql() {
- sql(null);
- }
-
public void sql(String sql) {
FlinkSqlExecutor.executeSql(sql, parameter, this);
}
@Override
public Table fromValues(Expression... values) {
- return delegate().fromValues(values);
+ return tableEnv.fromValues(values);
}
@Override
public Table fromValues(AbstractDataType<?> rowType, Expression... values) {
- return delegate().fromValues(rowType, values);
+ return tableEnv.fromValues(rowType, values);
}
@Override
public Table fromValues(Iterable<?> values) {
- return delegate().fromValues(values);
+ return tableEnv.fromValues(values);
}
@Override
public Table fromValues(AbstractDataType<?> rowType, Iterable<?> values) {
- return delegate().fromValues(rowType, values);
+ return tableEnv.fromValues(rowType, values);
}
@Override
- public void registerCatalog(String catalogName, Catalog catalog) {
- delegate().registerCatalog(catalogName, catalog);
+ public void createCatalog(String catalogName, CatalogDescriptor catalogDescriptor) {
+ tableEnv.createCatalog(catalogName, catalogDescriptor);
}
@Override
- public Optional<Catalog> getCatalog(String catalogName) {
- return delegate().getCatalog(catalogName);
+ public void useModules(String... moduleNames) {
+ tableEnv.useModules(moduleNames);
}
@Override
- public void loadModule(String moduleName, Module module) {
- delegate().loadModule(moduleName, module);
+ public void createFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createFunction(path, className, resourceUris);
}
@Override
- public void unloadModule(String moduleName) {
- delegate().unloadModule(moduleName);
+ public void createFunction(
+ String path,
+ String className,
+ List<ResourceUri> resourceUris,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
}
@Override
- public void createTemporarySystemFunction(
- String name, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createTemporarySystemFunction(name, functionClass);
+ public void createTemporaryFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporaryFunction(path, className, resourceUris);
}
@Override
- public void createTemporarySystemFunction(String name, UserDefinedFunction functionInstance) {
- delegate().createTemporarySystemFunction(name, functionInstance);
+ public void createTemporarySystemFunction(
+ String name, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporarySystemFunction(name, className, resourceUris);
}
@Override
- public boolean dropTemporarySystemFunction(String name) {
- return delegate().dropTemporarySystemFunction(name);
+ public void createTemporaryTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTemporaryTable(path, descriptor);
}
@Override
- public void createFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createFunction(path, functionClass);
+ public void createTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTable(path, descriptor);
}
@Override
- public void createFunction(
- String path,
- Class<? extends UserDefinedFunction> functionClass,
- boolean ignoreIfExists) {
- delegate().createFunction(path, functionClass, ignoreIfExists);
+ public Table from(TableDescriptor descriptor) {
+ return tableEnv.from(descriptor);
}
@Override
- public boolean dropFunction(String path) {
- return delegate().dropFunction(path);
+ public ModuleEntry[] listFullModules() {
+ return tableEnv.listFullModules();
}
@Override
- public void createTemporaryFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createTemporaryFunction(path, functionClass);
+ public String[] listTables(String catalogName, String databaseName) {
+ return tableEnv.listTables(catalogName, databaseName);
}
@Override
- public void createTemporaryFunction(String path, UserDefinedFunction functionInstance) {
- delegate().createTemporaryFunction(path, functionInstance);
+ public String explainSql(
+ String statement, ExplainFormat format, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, format, extraDetails);
}
@Override
- public boolean dropTemporaryFunction(String path) {
- return delegate().dropTemporaryFunction(path);
+ public CompiledPlan loadPlan(PlanReference planReference) throws TableException {
+ return tableEnv.loadPlan(planReference);
}
@Override
- public void createTemporaryView(String path, Table view) {
- delegate().createTemporaryView(path, view);
+ public CompiledPlan compilePlanSql(String statement) throws TableException {
+ return tableEnv.compilePlanSql(statement);
}
@Override
- public Table from(String path) {
- return delegate().from(path);
+ public void registerCatalog(String catalogName, Catalog catalog) {
+ tableEnv.registerCatalog(catalogName, catalog);
}
@Override
- public String[] listCatalogs() {
- return delegate().listCatalogs();
+ public Optional<Catalog> getCatalog(String catalogName) {
+ return tableEnv.getCatalog(catalogName);
}
@Override
- public String[] listModules() {
- return delegate().listModules();
+ public void loadModule(String moduleName, Module module) {
+ tableEnv.loadModule(moduleName, module);
}
@Override
- public String[] listDatabases() {
- return delegate().listDatabases();
+ public void unloadModule(String moduleName) {
+ tableEnv.unloadModule(moduleName);
}
@Override
- public String[] listTables() {
- return delegate().listTables();
+ public void createTemporarySystemFunction(
+ String name, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createTemporarySystemFunction(name, functionClass);
}
@Override
- public String[] listViews() {
- return delegate().listViews();
+ public void createTemporarySystemFunction(
+ String name, UserDefinedFunction functionInstance) {
+ tableEnv.createTemporarySystemFunction(name, functionInstance);
}
@Override
- public String[] listTemporaryTables() {
- return delegate().listTemporaryTables();
+ public boolean dropTemporarySystemFunction(String name) {
+ return tableEnv.dropTemporarySystemFunction(name);
}
@Override
- public String[] listTemporaryViews() {
- return delegate().listTemporaryViews();
+ public void createFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createFunction(path, functionClass);
}
@Override
- public String[] listUserDefinedFunctions() {
- return delegate().listUserDefinedFunctions();
+ public void createFunction(
+ String path,
+ Class<? extends UserDefinedFunction> functionClass,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, functionClass, ignoreIfExists);
}
@Override
- public String[] listFunctions() {
- return delegate().listFunctions();
+ public boolean dropFunction(String path) {
+ return tableEnv.dropFunction(path);
}
@Override
- public boolean dropTemporaryTable(String path) {
- return delegate().dropTemporaryTable(path);
+ public void createTemporaryFunction(
+ String path, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createTemporaryFunction(path, functionClass);
}
@Override
- public boolean dropTemporaryView(String path) {
- return delegate().dropTemporaryView(path);
+ public void createTemporaryFunction(String path, UserDefinedFunction functionInstance) {
+ tableEnv.createTemporaryFunction(path, functionInstance);
}
@Override
- public String explainSql(String statement, ExplainDetail... extraDetails) {
- return delegate().explainSql(statement, extraDetails);
+ public boolean dropTemporaryFunction(String path) {
+ return tableEnv.dropTemporaryFunction(path);
}
@Override
- public Table sqlQuery(String query) {
- return delegate().sqlQuery(query);
+ public void createTemporaryView(String path, Table view) {
+ tableEnv.createTemporaryView(path, view);
}
@Override
- public TableResult executeSql(String statement) {
- return delegate().executeSql(statement);
+ public Table from(String path) {
+ return tableEnv.from(path);
}
@Override
- public String getCurrentCatalog() {
- return delegate().getCurrentCatalog();
+ public String[] listCatalogs() {
+ return tableEnv.listCatalogs();
}
@Override
- public void useCatalog(String catalogName) {
- delegate().useCatalog(catalogName);
+ public String[] listModules() {
+ return tableEnv.listModules();
}
@Override
- public String getCurrentDatabase() {
- return delegate().getCurrentDatabase();
+ public String[] listDatabases() {
+ return tableEnv.listDatabases();
}
@Override
- public void useDatabase(String databaseName) {
- delegate().useDatabase(databaseName);
+ public String[] listTables() {
+ return tableEnv.listTables();
}
@Override
- public TableConfig getConfig() {
- return delegate().getConfig();
+ public String[] listViews() {
+ return tableEnv.listViews();
}
@Override
- public StatementSet createStatementSet() {
- return delegate().createStatementSet();
+ public String[] listTemporaryTables() {
+ return tableEnv.listTemporaryTables();
}
@Override
- public void useModules(String... modules) {
- delegate().useModules(modules);
+ public String[] listTemporaryViews() {
+ return tableEnv.listTemporaryViews();
}
@Override
- public ModuleEntry[] listFullModules() {
- return delegate().listFullModules();
+ public String[] listUserDefinedFunctions() {
+ return tableEnv.listUserDefinedFunctions();
}
@Override
- public void createTemporaryTable(String path, TableDescriptor descriptor) {
- delegate().createTemporaryTable(path, descriptor);
+ public String[] listFunctions() {
+ return tableEnv.listFunctions();
}
@Override
- public void createTable(String path, TableDescriptor descriptor) {
- delegate().createTable(path, descriptor);
+ public boolean dropTemporaryTable(String path) {
+ return tableEnv.dropTemporaryTable(path);
}
@Override
- public Table from(TableDescriptor descriptor) {
- return delegate().from(descriptor);
+ public boolean dropTemporaryView(String path) {
+ return tableEnv.dropTemporaryView(path);
}
@Override
- public void registerFunction(String name, ScalarFunction function) {
- delegate().registerFunction(name, function);
+ public String explainSql(String statement, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, extraDetails);
}
@Override
- public void registerTable(String name, Table table) {
- delegate().registerTable(name, table);
+ public Table sqlQuery(String query) {
+ return tableEnv.sqlQuery(query);
}
@Override
- public Table scan(String... tablePath) {
- return delegate().scan(tablePath);
+ public TableResult executeSql(String statement) {
+ return tableEnv.executeSql(statement);
}
@Override
- public String[] getCompletionHints(String statement, int position) {
- return delegate().getCompletionHints(statement, position);
+ public String getCurrentCatalog() {
+ return tableEnv.getCurrentCatalog();
}
@Override
- public String[] listTables(String catalogName, String databaseName) {
- return delegate().listTables(catalogName, databaseName);
+ public void useCatalog(String catalogName) {
+ tableEnv.useCatalog(catalogName);
}
@Override
- public CompiledPlan loadPlan(PlanReference planReference) {
- return delegate().loadPlan(planReference);
+ public String getCurrentDatabase() {
+ return tableEnv.getCurrentDatabase();
}
@Override
- public CompiledPlan compilePlanSql(String stmt) {
- return delegate().compilePlanSql(stmt);
+ public void useDatabase(String databaseName) {
+ tableEnv.useDatabase(databaseName);
}
@Override
- public void createFunction(String path, String className, List<ResourceUri> resourceUris) {
- delegate().createFunction(path, className, resourceUris);
+ public TableConfig getConfig() {
+ return tableEnv.getConfig();
}
@Override
- public void createFunction(
- String path, String className, List<ResourceUri> resourceUris, boolean ignoreIfExists) {
- delegate().createFunction(path, className, resourceUris, ignoreIfExists);
+ public StatementSet createStatementSet() {
+ return tableEnv.createStatementSet();
}
@Override
- public void createTemporaryFunction(String path, String className, List<ResourceUri> resourceUris) {
- delegate().createTemporaryFunction(path, className, resourceUris);
+ public String[] getCompletionHints(String statement, int position) {
+ return tableEnv.getCompletionHints(statement, position);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createTemporarySystemFunction(String name, String className, List<ResourceUri> resourceUris) {
- delegate().createTemporarySystemFunction(name, className, resourceUris);
+ public Table scan(String... tablePath) {
+ return tableEnv.scan(tablePath);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public String explainSql(String statement, ExplainFormat format, ExplainDetail... extraDetails) {
- return delegate().explainSql(statement, format, extraDetails);
+ public void registerTable(String name, Table table) {
+ tableEnv.registerTable(name, table);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createCatalog(String catalog, CatalogDescriptor catalogDescriptor) {
- delegate().createCatalog(catalog, catalogDescriptor);
+ public void registerFunction(String name, ScalarFunction function) {
+ tableEnv.registerFunction(name, function);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
similarity index 57%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
index fef182572..ddc5f5abf 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
@@ -20,14 +20,16 @@ package org.apache.streampark.flink.core;
import org.apache.commons.lang3.StringUtils;
import java.util.Optional;
-import java.util.function.Function;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-/** Supported Flink SQL command types. */
+/** Flink SQL command types. */
public enum SqlCommand {
+ // ---- SELECT Statements -----------------------------------------------------------------
SELECT("select", "(SELECT\\s+.+)"),
+
+ // ---- CREATE Statements -----------------------------------------------------------------
CREATE_TABLE("create table", "(CREATE\\s+(TEMPORARY\\s+|)TABLE\\s+.+)"),
CREATE_CATALOG("create catalog", "(CREATE\\s+CATALOG\\s+.+)"),
CREATE_DATABASE("create database", "(CREATE\\s+DATABASE\\s+.+)"),
@@ -37,26 +39,39 @@ public enum SqlCommand {
CREATE_FUNCTION(
"create function",
"(CREATE\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+(IF\\s+NOT\\s+EXISTS\\s+|)(\\S+)\\s+AS\\s+.*)"),
+
+ // ---- DROP Statements -------------------------------------------------------------------
DROP_CATALOG("drop catalog", "(DROP\\s+CATALOG\\s+.+)"),
DROP_TABLE("drop table", "(DROP\\s+(TEMPORARY\\s+|)TABLE\\s+.+)"),
DROP_DATABASE("drop database", "(DROP\\s+DATABASE\\s+.+)"),
DROP_VIEW("drop view", "(DROP\\s+(TEMPORARY\\s+|)VIEW\\s+.+)"),
DROP_FUNCTION(
- "drop function",
- "(DROP\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+ "drop function", "(DROP\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+
+ // ---- ALTER Statements ------------------------------------------------------------------
ALTER_TABLE("alter table", "(ALTER\\s+TABLE\\s+.+)"),
ALTER_VIEW("alter view", "(ALTER\\s+VIEW\\s+.+)"),
ALTER_DATABASE("alter database", "(ALTER\\s+DATABASE\\s+.+)"),
ALTER_FUNCTION(
"alter function",
"(ALTER\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+
+ // ---- INSERT Statement ------------------------------------------------------------------
INSERT("insert", "(INSERT\\s+(INTO|OVERWRITE)\\s+.+)"),
+
+ // ---- DESCRIBE Statement ----------------------------------------------------------------
DESC("desc", "(DESC\\s+.+)"),
DESCRIBE("describe", "(DESCRIBE\\s+.+)"),
+
+ // ---- EXPLAIN Statement -----------------------------------------------------------------
EXPLAIN("explain", "(EXPLAIN\\s+.+)"),
+
+ // ---- USE Statements --------------------------------------------------------------------
USE_CATALOG("use catalog", "(USE\\s+CATALOG\\s+.+)"),
USE_MODULES("use modules", "(USE\\s+MODULES\\s+.+)"),
USE_DATABASE("use database", "(USE\\s+(?!(CATALOG|MODULES)).+)"),
+
+ // ---- SHOW Statements -------------------------------------------------------------------
SHOW_CATALOGS("show catalogs", "(SHOW\\s+CATALOGS\\s*)"),
SHOW_CURRENT_CATALOG("show current catalog", "(SHOW\\s+CURRENT\\s+CATALOG\\s*)"),
SHOW_DATABASES("show databases", "(SHOW\\s+DATABASES\\s*)"),
@@ -68,65 +83,88 @@ public enum SqlCommand {
SHOW_CREATE_VIEW("show create view", "(SHOW\\s+CREATE\\s+VIEW\\s+.+)"),
SHOW_FUNCTIONS("show functions", "(SHOW\\s+(USER\\s+|)FUNCTIONS\\s*)"),
SHOW_MODULES("show modules", "(SHOW\\s+(FULL\\s+|)MODULES\\s*)"),
+
+ // ---- LOAD Statements -------------------------------------------------------------------
LOAD_MODULE("load module", "(LOAD\\s+MODULE\\s+.+)"),
+
+ // ---- UNLOAD Statements -----------------------------------------------------------------
UNLOAD_MODULE("unload module", "(UNLOAD\\s+MODULE\\s+.+)"),
- SET("set", "SET(\\s+(\\S+)\\s*=(.*))?", SqlCommandConverters::setOperands),
+
+ // ---- SET Statements --------------------------------------------------------------------
+ SET(
+ "set",
+ "SET(\\s+(\\S+)\\s*=(.*))?",
+ groups -> {
+ if (groups.length < 3) {
+ return Optional.empty();
+ }
+ if (groups[0] == null) {
+ return Optional.of(new String[]{cleanUp(groups[0])});
+ }
+ return Optional.of(new String[]{cleanUp(groups[1]), cleanUp(groups[2])});
+ }),
+
+ // ---- RESET Statements ------------------------------------------------------------------
RESET("reset", "RESET\\s+'(.*)'"),
- RESET_ALL("reset all", "RESET", SqlCommandConverters::resetAll),
+ RESET_ALL("reset all", "RESET", groups -> Optional.of(new String[]{"ALL"})),
+
+ // ---- INSERT SET Statements -------------------------------------------------------------
+ /** @deprecated SQL Client syntax; not supported on this platform. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
BEGIN_STATEMENT_SET(
- "begin statement set", "BEGIN\\s+STATEMENT\\s+SET", SqlCommandConverters::noOperands),
- END_STATEMENT_SET("end statement set", "END", SqlCommandConverters::noOperands),
+ "begin statement set", "BEGIN\\s+STATEMENT\\s+SET", SqlCommandConverters.NO_OPERANDS),
+ /** @deprecated SQL Client syntax; not supported on this platform. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
+ END_STATEMENT_SET("end statement set", "END", SqlCommandConverters.NO_OPERANDS),
+
+ // Since: 2.1.2 for flink 1.18
DELETE("delete", "(DELETE\\s+FROM\\s+.+)"),
UPDATE("update", "(UPDATE\\s+.+)");
- private final String commandName;
+ private static final int PATTERN_FLAGS = Pattern.CASE_INSENSITIVE | Pattern.DOTALL;
+
+ private final String name;
private final String regex;
- private final Function<String[], Optional<String[]>> converter;
+ private final SqlCommandConverter converter;
private Matcher matcher;
- SqlCommand(String commandName, String regex) {
- this(commandName, regex, SqlCommandConverters::firstGroup);
+ SqlCommand(String name, String regex) {
+ this(name, regex, SqlCommandConverters.DEFAULT);
}
- SqlCommand(
- String commandName,
- String regex,
- Function<String[], Optional<String[]>> converter) {
- this.commandName = commandName;
+ SqlCommand(String name, String regex, SqlCommandConverter converter) {
+ this.name = name;
this.regex = regex;
this.converter = converter;
}
- /** Scala field-style access (lowercase command label). */
- public String commandName() {
- return commandName;
+ /** Command label (e.g. {@code "select"}, {@code "create table"}). */
+ public String getName() {
+ return name;
}
- public String getCommandName() {
- return commandName;
+ public String getRegex() {
+ return regex;
}
- public Matcher matcher() {
- return matcher;
+ public SqlCommandConverter getConverter() {
+ return converter;
}
public Matcher getMatcher() {
return matcher;
}
- public Optional<String[]> convert(String[] groups) {
- return converter.apply(groups);
- }
-
public boolean matches(String input) {
if (StringUtils.isBlank(regex)) {
return false;
}
- Pattern pattern = Pattern.compile(regex, Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
+ Pattern pattern = Pattern.compile(regex, PATTERN_FLAGS);
matcher = pattern.matcher(input);
return matcher.matches();
}
+ /** Resolve the first matching command for the given statement. */
public static SqlCommand get(String stmt) {
for (SqlCommand command : values()) {
if (command.matches(stmt)) {
@@ -135,4 +173,8 @@ public enum SqlCommand {
}
return null;
}
+
+ static String cleanUp(String sql) {
+ return sql.trim().replaceAll("^(['\"])|(['\"])$", "");
+ }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/StreamEnvConfigFunction.java
similarity index 62%
rename from streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
rename to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/StreamEnvConfigFunction.java
index 4914c46e9..d1ed71bb3 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/StreamEnvConfigFunction.java
@@ -17,11 +17,18 @@
package org.apache.streampark.flink.core;
-import org.apache.flink.client.program.ClusterClient;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.util.ParameterTool;
-public class FlinkClusterClient<T> extends FlinkClientTrait<T> {
+@FunctionalInterface
+public interface StreamEnvConfigFunction {
- public FlinkClusterClient(ClusterClient<T> clusterClient) {
- super(clusterClient);
- }
+ /**
+ * When used to initialize StreamExecutionEnvironment, it can be used to implement this function
+ * and customize the parameters to be set...
+ *
+ * @param environment
+ * @param parameterTool
+ */
+ void configuration(StreamExecutionEnvironment environment, ParameterTool parameterTool);
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/TableEnvConfigFunction.java
similarity index 65%
rename from streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java
rename to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/TableEnvConfigFunction.java
index e300de2a6..0e57b74b5 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/TableEnvConfigFunction.java
@@ -17,11 +17,18 @@
package org.apache.streampark.flink.core;
-import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.util.ParameterTool;
-public class FlinkKubernetesClient extends FlinkKubernetesClientTrait {
+@FunctionalInterface
+public interface TableEnvConfigFunction {
- public FlinkKubernetesClient(FlinkKubeClient kubeClient) {
- super(kubeClient);
- }
+ /**
+ * When used to initialize the TableEnvironment, it can be used to implement this function and
+ * customize the parameters to be set...
+ *
+ * @param tableConfig
+ * @param parameterTool
+ */
+ void configuration(TableConfig tableConfig, ParameterTool parameterTool);
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
similarity index 54%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
index f269d77df..c55457631 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
@@ -17,14 +17,15 @@
package org.apache.streampark.flink.core.conf;
-import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.ParameterTool;
-public final class FlinkConfiguration {
+/** Flink runtime configuration holder. */
+public class FlinkConfiguration {
- private final ParameterTool parameter;
- private final Configuration envConfig;
- private final Configuration tableConfig;
+ public final ParameterTool parameter;
+ public final Configuration envConfig;
+ public final Configuration tableConfig;
public FlinkConfiguration(
ParameterTool parameter, Configuration envConfig, Configuration tableConfig) {
@@ -33,34 +34,7 @@ public final class FlinkConfiguration {
this.tableConfig = tableConfig;
}
- public ParameterTool getParameter() {
- return parameter;
- }
-
- public Configuration getEnvConfig() {
- return envConfig;
- }
-
- public Configuration getTableConfig() {
- return tableConfig;
- }
-
- /** Scala API alias for {@link #getParameter()}. */
- public ParameterTool parameter() {
- return parameter;
- }
-
- /** Scala API alias for {@link #getEnvConfig()}. */
- public Configuration envConfig() {
- return envConfig;
- }
-
- /** Scala API alias for {@link #getTableConfig()}. */
- public Configuration tableConfig() {
- return tableConfig;
- }
-
- public FlinkConfiguration withParameter(ParameterTool newParameter) {
- return new FlinkConfiguration(newParameter, envConfig, tableConfig);
+ public FlinkConfiguration withParameter(ParameterTool parameter) {
+ return new FlinkConfiguration(parameter, envConfig, tableConfig);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
similarity index 61%
copy from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
copy to streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
index 2df697fa8..441dbf3d7 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
@@ -17,18 +17,39 @@
package org.apache.streampark.flink.util;
-import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.util.TimeUtils;
import java.io.File;
import java.time.Duration;
+import java.util.Arrays;
import java.util.Map;
+import java.util.stream.Collectors;
+/** Flink utility methods. */
public final class FlinkUtils {
private FlinkUtils() {
}
+ public static <R> ListState<R> getUnionListState(
+ FunctionInitializationContext context,
+ String descriptorName,
+ TypeInformation<R> typeInformation) {
+ try {
+ return context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ descriptorName, typeInformation.getTypeClass()));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
public static String getFlinkDistJar(String flinkHome) {
String[] jars =
new File(flinkHome + "/lib")
@@ -44,14 +65,15 @@ public final class FlinkUtils {
"[StreamPark] found multiple flink-dist jar in "
+ flinkHome
+ "/lib,["
- + String.join(",", jars)
+ + Arrays.stream(jars).collect(Collectors.joining(","))
+ "]");
}
public static boolean isCheckpointEnabled(Map<String, String> map) {
Duration checkpointInterval =
TimeUtils.parseDuration(
- map.getOrDefault(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL.key(), "0ms"));
+ map.getOrDefault(
+ CheckpointingOptions.CHECKPOINTING_INTERVAL.key(), "0ms"));
return checkpointInterval.toMillis() > 0;
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/test/java/org/apache/streampark/RegExpTest.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/test/java/org/apache/streampark/RegExpTest.java
new file mode 100644
index 000000000..dc7e648e1
--- /dev/null
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base-v2/src/test/java/org/apache/streampark/RegExpTest.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** some simple tests */
+class RegExpTest {
+
+ /**
+ * Case insensitive, matches everything as one line, . matches newlines, note: \s matches any
+ * whitespace character, including newlines
+ */
+ public static final int DEFAULT_PATTERN_FLAGS = Pattern.CASE_INSENSITIVE | Pattern.DOTALL;
+
+ /**
+ * CREATE CATALOG catalog_name WITH (key1=val1, key2=val2, ...)<br>
+ * Example：create catalog hive_catalog with('name' = 'my_hive', 'conf' = '/home/hive/conf')
+ */
+ private static final Pattern CREATE_HIVE_CATALOG =
+ Pattern.compile("CREATE\\s+CATALOG\\s+\\S[\\s\\S]*", DEFAULT_PATTERN_FLAGS);
+
+ @Test
+ void testCreateHiveCatalog() {
+ String str = "create catalog hive with (\n"
+ + " 'type' = 'hive',\n"
+ + " 'hadoop-conf-dir' = 'D:\\IDEAWorkspace\\work\\baishan\\log\\data-max\\src\\main\\resources',\n"
+ + " 'hive-conf-dir' = 'D:\\IDEAWorkspace\\work\\baishan\\log\\data-max\\src\\main\\resources'\n"
+ + ")";
+ Matcher matcher = CREATE_HIVE_CATALOG.matcher(str);
+ System.out.println(matcher.matches());
+ }
+
+ /**
+ * CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF NOT EXISTS]
+ * [catalog_name.][db_name.]function_name AS identifier [LANGUAGE JAVA|SCALA|PYTHON]<br>
+ * Example：create function test_fun as com.flink.testFun
+ */
+ private static final Pattern CREATE_FUNCTION =
+ Pattern.compile(
+ "CREATE\\s+(?:TEMPORARY\\s+(?:SYSTEM\\s+)?)?FUNCTION\\s+(?:IF NOT EXISTS\\s+)?"
+ + "([A-Za-z][A-Za-z\\d.\\-_]*)\\s+AS\\s+'([^']+)'\\s+LANGUAGE\\s+(JAVA|SCALA|PYTHON)",
+ DEFAULT_PATTERN_FLAGS);
+
+ @Test
+ void testCreateFunction() {
+ String str =
+ "create function if not exists hive.get_json_value as 'com.flink.function.JsonValueFunction' language java";
+ Matcher matcher = CREATE_FUNCTION.matcher(str);
+ System.out.println(matcher.matches());
+ }
+
+ /** USE [catalog_name.]database_name */
+ private static final Pattern USE_DATABASE =
+ Pattern.compile("USE\\s+(?!(?:CATALOG|MODULES)\\b)\\S[\\s\\S]*", DEFAULT_PATTERN_FLAGS);
+
+ @Test
+ void testUseDatabase() {
+ String str = "use modul.a ";
+ Matcher matcher = USE_DATABASE.matcher(str);
+ System.out.println(matcher.matches());
+ }
+
+ /** SHOW [USER] FUNCTIONS */
+ private static final Pattern SHOW_FUNCTIONS =
+ Pattern.compile("SHOW\\s+(?:USER\\s+)?FUNCTIONS\\b", DEFAULT_PATTERN_FLAGS);
+
+ @Test
+ void testShowFunction() {
+ String str = "show user functions";
+ Matcher matcher = SHOW_FUNCTIONS.matcher(str);
+ System.out.println(matcher.matches());
+ }
+}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/pom.xml b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/pom.xml
index c5647870a..c7734c6d8 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/pom.xml
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/pom.xml
@@ -25,14 +25,14 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-shims-base_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-base</artifactId>
<name>StreamPark : Flink Shims Base</name>
<dependencies>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-common_${scala.binary.version}</artifactId>
+ <artifactId>streampark-common</artifactId>
</dependency>
<!--flink base-->
@@ -43,13 +43,6 @@
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-scala_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
@@ -64,20 +57,6 @@
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java</artifactId>
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkClientTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkClientTrait.java
index a920be68e..f7ae7357e 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkClientTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkClientTrait.java
@@ -23,6 +23,7 @@ import org.apache.flink.core.execution.SavepointFormatType;
import java.util.concurrent.CompletableFuture;
+/** Flink cluster client savepoint operations. */
public abstract class FlinkClientTrait<T> {
protected final ClusterClient<T> clusterClient;
@@ -37,9 +38,10 @@ public abstract class FlinkClientTrait<T> {
public CompletableFuture<String> triggerSavepoint(
JobID jobID, String savepointDir, boolean nativeFormat) {
- SavepointFormatType formatType =
- nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT;
- return clusterClient.triggerSavepoint(jobID, savepointDir, formatType);
+ return clusterClient.triggerSavepoint(
+ jobID,
+ savepointDir,
+ nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT);
}
public CompletableFuture<String> cancelWithSavepoint(JobID jobID, String savepointDir) {
@@ -48,9 +50,10 @@ public abstract class FlinkClientTrait<T> {
public CompletableFuture<String> cancelWithSavepoint(
JobID jobID, String savepointDir, boolean nativeFormat) {
- SavepointFormatType formatType =
- nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT;
- return clusterClient.cancelWithSavepoint(jobID, savepointDir, formatType);
+ return clusterClient.cancelWithSavepoint(
+ jobID,
+ savepointDir,
+ nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT);
}
public CompletableFuture<String> stopWithSavepoint(
@@ -64,9 +67,10 @@ public abstract class FlinkClientTrait<T> {
boolean advanceToEndOfEventTime,
String savepointDir,
boolean nativeFormat) {
- SavepointFormatType formatType =
- nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT;
return clusterClient.stopWithSavepoint(
- jobID, advanceToEndOfEventTime, savepointDir, formatType);
+ jobID,
+ advanceToEndOfEventTime,
+ savepointDir,
+ nativeFormat ? SavepointFormatType.NATIVE : SavepointFormatType.DEFAULT);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClientTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClientTrait.java
index e40f0780f..02cbd0a56 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClientTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClientTrait.java
@@ -22,6 +22,7 @@ import org.apache.flink.kubernetes.kubeclient.resources.KubernetesService;
import java.util.Optional;
+/** Flink Kubernetes client operations. */
public abstract class FlinkKubernetesClientTrait {
protected final FlinkKubeClient kubeClient;
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
similarity index 63%
rename from streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java
rename to streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
index 5ed66ed7a..d1868525c 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkEnvironmentUtils.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkParameterUtils.java
@@ -18,42 +18,57 @@
package org.apache.streampark.flink.core;
import org.apache.streampark.common.conf.ConfigKeys;
+import org.apache.streampark.common.util.AssertUtils;
import org.apache.streampark.common.util.DeflaterUtils;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.PipelineOptions;
import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
-public final class FlinkEnvironmentUtils {
+import javax.annotation.Nullable;
- private FlinkEnvironmentUtils() {
+/** Parameter and table environment helpers for Flink application naming. */
+public final class FlinkParameterUtils {
+
+ private FlinkParameterUtils() {
}
public static String getAppName(ParameterTool parameterTool) {
return getAppName(parameterTool, null, false);
}
- public static String getAppName(ParameterTool parameterTool, String name, boolean required) {
- String appName = name;
- if (appName == null) {
- try {
- appName =
- DeflaterUtils.unzipString(parameterTool.get(ConfigKeys.KEY_APP_NAME(), null));
- } catch (Exception ignored) {
- // match Scala Try(...).getOrElse(...)
+ public static String getAppName(ParameterTool parameterTool, boolean required) {
+ return getAppName(parameterTool, null, required);
+ }
+
+ public static String getAppName(
+ ParameterTool parameterTool, @Nullable String name, boolean required) {
+ String appName;
+ if (name == null) {
+ appName = null;
+ String zippedAppName = parameterTool.get(ConfigKeys.KEY_APP_NAME(), null);
+ if (zippedAppName != null) {
+ try {
+ appName = DeflaterUtils.unzipString(zippedAppName);
+ } catch (Exception ignored) {
+ // fall back to pipeline.name
+ }
}
if (appName == null) {
appName = parameterTool.get(ConfigKeys.KEY_FLINK_APP_NAME(), null);
}
+ } else {
+ appName = name;
}
- if (required && appName == null) {
- throw new IllegalArgumentException("[StreamPark] Application name cannot be null");
+ if (required) {
+ AssertUtils.required(
+ appName != null, "[StreamPark] Application name cannot be null");
}
return appName;
}
- public static TableEnvironment setAppName(TableEnvironment env, ParameterTool parameter) {
+ public static <T extends TableEnvironment> T setAppName(T env, ParameterTool parameter) {
String appName = getAppName(parameter);
if (appName != null) {
env.getConfig().getConfiguration().setString(PipelineOptions.NAME, appName);
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java
index 0fabead7d..278f40a5b 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlExecutor.java
@@ -19,28 +19,39 @@ package org.apache.streampark.flink.core;
import org.apache.streampark.common.conf.ConfigKeys;
import org.apache.streampark.common.util.AssertUtils;
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
+
+import org.apache.streampark.shaded.org.slf4j.Logger;
import org.apache.commons.lang3.StringUtils;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.ExecutionOptions;
-import org.apache.flink.table.api.StatementSet;
import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
-import java.lang.reflect.Field;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.function.Consumer;
+import java.util.stream.Collectors;
-/** Executes parsed Flink SQL statements against a table environment. */
+/** Executes Flink SQL statements. */
public final class FlinkSqlExecutor {
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory()
+ .getLogger(FlinkSqlExecutor.class.getName());
+
private static final ReentrantReadWriteLock.WriteLock LOCK =
new ReentrantReadWriteLock().writeLock();
- private static final Log LOG = new Log();
+ private static final Map<SqlCommand, CommandHandler> COMMAND_HANDLERS = buildCommandHandlers();
private FlinkSqlExecutor() {
}
@@ -49,32 +60,48 @@ public final class FlinkSqlExecutor {
executeSql(sql, parameter, context, null);
}
- static void executeSql(
- String sql,
- ParameterTool parameter,
- TableEnvironment context,
- Consumer<String> callbackFunc) {
+ public static void executeSql(
+ String sql,
+ ParameterTool parameter,
+ TableEnvironment context,
+ Consumer<String> callbackFunc) {
+ String flinkSql = resolveSql(sql, parameter);
+ ExecutionContext ctx = new ExecutionContext(context, callbackFunc, parameter);
+ List<SqlCommandCall> calls = SqlCommandParser.parseSQL(flinkSql, null);
+ for (SqlCommandCall call : calls) {
+ processCommand(call, ctx);
+ }
+ finishExecution(flinkSql, ctx);
+ }
+
+ private static String resolveSql(String sql, ParameterTool parameter) {
String flinkSql =
- StringUtils.isBlank(sql) ? parameter.get(ConfigKeys.KEY_FLINK_SQL()) : parameter.get(sql);
+ StringUtils.isBlank(sql)
+ ? parameter.get(ConfigKeys.KEY_FLINK_SQL())
+ : parameter.get(sql);
if (StringUtils.isBlank(flinkSql)) {
throw new IllegalArgumentException("verify failed: flink sql cannot be empty");
}
+ return flinkSql;
+ }
- String runMode = parameter.get(ExecutionOptions.RUNTIME_MODE.key());
- boolean hasInsert = false;
- StatementSet statementSet = context.createStatementSet();
- List<SqlCommandCall> commands = SqlCommandParser.parseSQL(flinkSql);
-
- for (SqlCommandCall call : commands) {
- if (handleCommand(call, context, statementSet, callbackFunc, runMode)) {
- hasInsert = true;
- }
- }
+ private static void processCommand(SqlCommandCall call, ExecutionContext ctx) {
+ COMMAND_HANDLERS.getOrDefault(call.command, FlinkSqlExecutor::executeDefault).handle(call, ctx);
+ }
- if (hasInsert) {
- TableResult result = statementSet.execute();
- if (result != null && result.getJobClient().isPresent()) {
- LOG.info("jobId:" + result.getJobClient().get().getJobID());
+ private static void finishExecution(String flinkSql, ExecutionContext ctx) {
+ if (ctx.hasInsert) {
+ TableResult result = ctx.statementSet.execute();
+ if (result != null) {
+ result.getJobClient()
+ .ifPresent(
+ jobClient -> {
+ try {
+ LOG.info("jobId:{}", jobClient.getJobID());
+ } catch (Exception ignored) {
+ // ignore
+ }
+ });
}
} else {
LOG.error("No 'INSERT' statement to trigger the execution of the Flink job.");
@@ -83,174 +110,184 @@ public final class FlinkSqlExecutor {
}
LOG.info(
- "\n\n\n==============flinkSql==============\n\n " + flinkSql + "\n\n============================\n\n\n");
+ "\n\n\n==============flinkSql==============\n\n {}\n\n============================\n\n\n",
+ flinkSql);
}
- /**
- * @return {@code true} when an INSERT statement was added to the statement set
- */
- private static boolean handleCommand(
- SqlCommandCall call,
- TableEnvironment context,
- StatementSet statementSet,
- Consumer<String> callbackFunc,
- String runMode) {
- String args = call.operands().length == 0 ? null : call.operands()[0];
- String command = call.command().getCommandName();
- switch (call.command()) {
- case SHOW_CATALOGS:
- callback(callbackFunc, command + ": " + String.join("\n", context.listCatalogs()));
- return false;
- case SHOW_CURRENT_CATALOG:
- callback(callbackFunc, command + ": " + context.getCurrentCatalog());
- return false;
- case SHOW_DATABASES:
- callback(callbackFunc, command + ": " + String.join("\n", context.listDatabases()));
- return false;
- case SHOW_CURRENT_DATABASE:
- callback(callbackFunc, command + ": " + context.getCurrentDatabase());
- return false;
- case SHOW_TABLES:
- callback(callbackFunc, command + ": " + listVisibleTables(context));
- return false;
- case SHOW_FUNCTIONS:
- callback(
- callbackFunc,
- command + ": " + String.join("\n", context.listUserDefinedFunctions()));
- return false;
- case SHOW_MODULES:
- callback(callbackFunc, command + ": " + String.join("\n", context.listModules()));
- return false;
- case DESC:
- case DESCRIBE:
- callback(callbackFunc, describeTable(context, args));
- return false;
- case EXPLAIN:
- TableResult tableResult = context.executeSql(call.originSql());
- String explain = tableResult.collect().next().getField(0).toString();
- callback(callbackFunc, explain);
- return false;
- case SET:
- String operand = call.operands()[1];
- LOG.info(command + ": " + args + " --> " + operand);
- context.getConfig().getConfiguration().setString(args, operand);
- return false;
- case RESET:
- case RESET_ALL:
- resetConfiguration(context, call.command(), args);
- LOG.info(command + ": " + args);
- return false;
- case BEGIN_STATEMENT_SET:
- case END_STATEMENT_SET:
- LOG.warn("SQL Client Syntax: " + call.command().getCommandName());
- return false;
- case INSERT:
- statementSet.addInsertSql(call.originSql());
- return true;
- case SELECT:
- LOG.error("StreamPark dose not support 'SELECT' statement now!");
- throw new UnsupportedOperationException(
- "StreamPark dose not support 'select' statement now!");
- case DELETE:
- case UPDATE:
- AssertUtils.required(
- !"STREAMING".equals(runMode),
- "Currently, "
- + command.toUpperCase()
- + " statement only supports in batch mode, "
- + "and it requires the target table connector implements the SupportsRowLevelDelete, "
- + "For more details please refer to: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/"
- + command.toLowerCase());
- return false;
- default:
- executeDefaultSql(context, call, command, args);
- return false;
- }
+ private static Map<SqlCommand, CommandHandler> buildCommandHandlers() {
+ Map<SqlCommand, CommandHandler> handlers = new EnumMap<>(SqlCommand.class);
+ handlers.put(SqlCommand.SHOW_CATALOGS,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listCatalogs())));
+ handlers.put(
+ SqlCommand.SHOW_CURRENT_CATALOG,
+ (call, ctx) -> showMeta(call, ctx, ctx.context.getCurrentCatalog()));
+ handlers.put(
+ SqlCommand.SHOW_DATABASES,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listDatabases())));
+ handlers.put(
+ SqlCommand.SHOW_CURRENT_DATABASE,
+ (call, ctx) -> showMeta(call, ctx, ctx.context.getCurrentDatabase()));
+ handlers.put(SqlCommand.SHOW_TABLES, FlinkSqlExecutor::showTables);
+ handlers.put(
+ SqlCommand.SHOW_FUNCTIONS,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listUserDefinedFunctions())));
+ handlers.put(
+ SqlCommand.SHOW_MODULES,
+ (call, ctx) -> showMeta(call, ctx, joinLines(ctx.context.listModules())));
+ handlers.put(SqlCommand.DESC, FlinkSqlExecutor::describeTable);
+ handlers.put(SqlCommand.DESCRIBE, FlinkSqlExecutor::describeTable);
+ handlers.put(SqlCommand.EXPLAIN, FlinkSqlExecutor::explainSql);
+ handlers.put(SqlCommand.SET, FlinkSqlExecutor::setConfig);
+ handlers.put(SqlCommand.RESET, FlinkSqlExecutor::resetConfig);
+ handlers.put(SqlCommand.RESET_ALL, FlinkSqlExecutor::resetConfig);
+ handlers.put(SqlCommand.BEGIN_STATEMENT_SET, FlinkSqlExecutor::warnStatementSet);
+ handlers.put(SqlCommand.END_STATEMENT_SET, FlinkSqlExecutor::warnStatementSet);
+ handlers.put(SqlCommand.INSERT, FlinkSqlExecutor::addInsert);
+ handlers.put(SqlCommand.SELECT, FlinkSqlExecutor::rejectSelect);
+ handlers.put(SqlCommand.DELETE, FlinkSqlExecutor::validateBatchCommand);
+ handlers.put(SqlCommand.UPDATE, FlinkSqlExecutor::validateBatchCommand);
+ return handlers;
}
- private static String listVisibleTables(TableEnvironment context) {
- StringBuilder tables = new StringBuilder();
- for (String table : context.listTables()) {
- if (!table.startsWith("UnnamedTable")) {
- if (tables.length() > 0) {
- tables.append('\n');
- }
- tables.append(table);
- }
- }
- return tables.toString();
+ private static void showMeta(SqlCommandCall call, ExecutionContext ctx, String payload) {
+ ctx.callback.accept(call.command.getName() + ": " + payload);
+ }
+
+ private static void showTables(SqlCommandCall call, ExecutionContext ctx) {
+ String tables =
+ Arrays.stream(ctx.context.listTables())
+ .filter(t -> !t.startsWith("UnnamedTable"))
+ .collect(Collectors.joining("\n"));
+ showMeta(call, ctx, tables);
}
- private static String describeTable(TableEnvironment context, String args) {
- var schema = context.scan(args).getSchema();
- StringBuilder builder = new StringBuilder("Column\tType\n");
+ private static void describeTable(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
+ TableSchema schema = ctx.context.scan(args).getSchema();
+ StringBuilder builder = new StringBuilder();
+ builder.append("Column\tType\n");
for (int i = 0; i <= schema.getFieldCount(); i++) {
- builder
- .append(schema.getFieldName(i).get())
- .append('\t')
+ builder.append(schema.getFieldName(i).get())
+ .append("\t")
.append(schema.getFieldDataType(i).get())
- .append('\n');
+ .append("\n");
}
- return builder.toString();
+ ctx.callback.accept(builder.toString());
}
- private static void executeDefaultSql(
- TableEnvironment context,
- SqlCommandCall call,
- String command,
- String args) {
- LOCK.lock();
- try {
- context.executeSql(call.originSql());
- LOG.info(command + ":" + args);
- } finally {
- if (LOCK.isHeldByCurrentThread()) {
- LOCK.unlock();
- }
- }
+ private static void explainSql(SqlCommandCall call, ExecutionContext ctx) {
+ TableResult tableResult = ctx.context.executeSql(call.originSql);
+ Row row = tableResult.collect().next();
+ ctx.callback.accept(row.getField(0).toString());
}
- private static void callback(Consumer<String> callbackFunc, String message) {
- if (callbackFunc == null) {
- LOG.info(message);
- } else {
- callbackFunc.accept(message);
- }
+ private static void setConfig(SqlCommandCall call, ExecutionContext ctx) {
+ AssertUtils.required(
+ call.operands != null && call.operands.length >= 2,
+ "SET command requires key and value operands");
+ String args = call.operands[0];
+ String operand = call.operands[1];
+ LOG.info("{}: {} --> {}", call.command.getName(), args, operand);
+ ctx.context.getConfig().getConfiguration().setString(args, operand);
}
- private static void resetConfiguration(TableEnvironment context, SqlCommand command, String args) {
+ private static void resetConfig(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
try {
- Field confDataField = Configuration.class.getDeclaredField("confData");
+ java.lang.reflect.Field confDataField =
+ Configuration.class.getDeclaredField("confData");
confDataField.setAccessible(true);
- Object confDataObject = confDataField.get(context.getConfig().getConfiguration());
- if (!(confDataObject instanceof Map)) {
- throw new IllegalStateException("Unexpected Flink configuration internal structure");
- }
- Map<?, ?> confData = (Map<?, ?>) confDataObject;
+ @SuppressWarnings("unchecked")
+ HashMap<String, Object> confData =
+ (HashMap<String, Object>) confDataField.get(ctx.context.getConfig().getConfiguration());
synchronized (confData) {
- if (command == SqlCommand.RESET) {
+ if (call.command == SqlCommand.RESET) {
confData.remove(args);
} else {
confData.clear();
}
}
+ LOG.info("{}: {}", call.command.getName(), args);
} catch (ReflectiveOperationException e) {
- throw new IllegalStateException("Failed to reset table configuration", e);
+ throw new IllegalStateException("Failed to reset Flink table configuration", e);
}
}
- private static final class Log extends org.apache.streampark.common.util.LoggerSupport {
+ private static void warnStatementSet(SqlCommandCall call, ExecutionContext ctx) {
+ LOG.warn("SQL Client Syntax: {} ", call.command.getName());
+ }
- void info(String msg) {
- logInfo(msg);
- }
+ private static void addInsert(SqlCommandCall call, ExecutionContext ctx) {
+ ctx.statementSet.addInsertSql(call.originSql);
+ ctx.hasInsert = true;
+ }
+
+ private static void rejectSelect(SqlCommandCall call, ExecutionContext ctx) {
+ LOG.error("StreamPark dose not support 'SELECT' statement now!");
+ throw new UnsupportedOperationException("StreamPark dose not support 'select' statement now!");
+ }
- void warn(String msg) {
- logWarn(msg);
+ private static void validateBatchCommand(SqlCommandCall call, ExecutionContext ctx) {
+ String runMode = ctx.parameter.get(ExecutionOptions.RUNTIME_MODE.key());
+ AssertUtils.required(
+ !"STREAMING".equals(runMode),
+ "Currently, "
+ + call.command.getName().toUpperCase()
+ + " statement only supports in batch mode, "
+ + "and it requires the target table connector implements the SupportsRowLevelDelete, "
+ + "For more details please refer to: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/"
+ + call.command.getName());
+ }
+
+ private static void executeDefault(SqlCommandCall call, ExecutionContext ctx) {
+ String args = firstOperand(call);
+ try {
+ LOCK.lock();
+ ctx.context.executeSql(call.originSql);
+ LOG.info("{}:{}", call.command.getName(), args);
+ } finally {
+ if (LOCK.isHeldByCurrentThread()) {
+ LOCK.unlock();
+ }
}
+ }
+
+ private static String firstOperand(SqlCommandCall call) {
+ return call.operands.length == 0 ? null : call.operands[0];
+ }
+
+ private static String joinLines(String[] values) {
+ return String.join("\n", values);
+ }
+
+ @FunctionalInterface
+ private interface CommandHandler {
+
+ void handle(SqlCommandCall call, ExecutionContext ctx);
+ }
+
+ private static final class ExecutionContext {
+
+ private final TableEnvironment context;
+ private final Consumer<String> callback;
+ private final ParameterTool parameter;
+ private final org.apache.flink.table.api.StatementSet statementSet;
+ private boolean hasInsert;
- void error(String msg) {
- logError(msg);
+ private ExecutionContext(
+ TableEnvironment context, Consumer<String> callbackFunc,
+ ParameterTool parameter) {
+ this.context = context;
+ this.parameter = parameter;
+ this.statementSet = context.createStatementSet();
+ this.callback =
+ r -> {
+ if (callbackFunc != null) {
+ callbackFunc.accept(r);
+ } else {
+ LOG.info(r);
+ }
+ };
}
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidationResult.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidationResult.java
index 0008b5df0..3304f5178 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidationResult.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidationResult.java
@@ -19,50 +19,39 @@ package org.apache.streampark.flink.core;
import org.apache.streampark.common.enums.FlinkSqlValidationFailedType;
-/** Result of Flink SQL validation (Scala case-class compatible). */
+/** Flink SQL validation result. */
public class FlinkSqlValidationResult {
- private boolean success = true;
- private FlinkSqlValidationFailedType failedType;
- private int lineStart;
- private int lineEnd;
- private int errorLine;
- private int errorColumn;
- private String sql;
- private String exception;
-
- /** Default success result; mirrors the Scala case-class no-arg constructor. */
- public FlinkSqlValidationResult() {
- // Intentionally empty: fields default to a successful validation result.
+ public final boolean success;
+ public final FlinkSqlValidationFailedType failedType;
+ public final int lineStart;
+ public final int lineEnd;
+ public final int errorLine;
+ public final int errorColumn;
+ public final String sql;
+ public final String exception;
+
+ private FlinkSqlValidationResult(Builder builder) {
+ this.success = builder.success;
+ this.failedType = builder.failedType;
+ this.lineStart = builder.lineStart;
+ this.lineEnd = builder.lineEnd;
+ this.errorLine = builder.errorLine;
+ this.errorColumn = builder.errorColumn;
+ this.sql = builder.sql;
+ this.exception = builder.exception;
}
- /** Builds a failed validation result without exposing an oversized public constructor. */
- public static FlinkSqlValidationResult failure(
- FlinkSqlValidationFailedType failedType,
- int lineStart,
- int lineEnd,
- int errorLine,
- int errorColumn,
- String sql,
- String exception) {
- FlinkSqlValidationResult result = new FlinkSqlValidationResult();
- result.success = false;
- result.failedType = failedType;
- result.lineStart = lineStart;
- result.lineEnd = lineEnd;
- result.errorLine = errorLine;
- result.errorColumn = errorColumn;
- result.sql = sql;
- result.exception = exception;
- return result;
+ public static Builder builder() {
+ return new Builder();
}
- /** Scala field-style access. */
- public boolean success() {
- return success;
+ /** Create a successful validation result with default values. */
+ public static FlinkSqlValidationResult ok() {
+ return builder().build();
}
- public boolean isSuccess() {
+ public boolean success() {
return success;
}
@@ -70,95 +59,84 @@ public class FlinkSqlValidationResult {
return failedType;
}
- public FlinkSqlValidationFailedType getFailedType() {
- return failedType;
- }
-
public int lineStart() {
return lineStart;
}
- public int getLineStart() {
- return lineStart;
- }
-
public int lineEnd() {
return lineEnd;
}
- public int getLineEnd() {
- return lineEnd;
- }
-
public int errorLine() {
return errorLine;
}
- public int getErrorLine() {
- return errorLine;
- }
-
public int errorColumn() {
return errorColumn;
}
- public int getErrorColumn() {
- return errorColumn;
- }
-
public String sql() {
return sql;
}
- public String getSql() {
- return sql;
- }
-
public String exception() {
return exception;
}
- public String getException() {
- return exception;
- }
-
- public FlinkSqlValidationResult withSuccess(boolean success) {
- this.success = success;
- return this;
- }
-
- public FlinkSqlValidationResult withFailedType(FlinkSqlValidationFailedType failedType) {
- this.failedType = failedType;
- return this;
- }
-
- public FlinkSqlValidationResult withLineStart(int lineStart) {
- this.lineStart = lineStart;
- return this;
- }
-
- public FlinkSqlValidationResult withLineEnd(int lineEnd) {
- this.lineEnd = lineEnd;
- return this;
- }
-
- public FlinkSqlValidationResult withErrorLine(int errorLine) {
- this.errorLine = errorLine;
- return this;
- }
-
- public FlinkSqlValidationResult withErrorColumn(int errorColumn) {
- this.errorColumn = errorColumn;
- return this;
- }
-
- public FlinkSqlValidationResult withSql(String sql) {
- this.sql = sql;
- return this;
- }
-
- public FlinkSqlValidationResult withException(String exception) {
- this.exception = exception;
- return this;
+ /** Builder for {@link FlinkSqlValidationResult}. */
+ public static class Builder {
+
+ private boolean success = true;
+ private FlinkSqlValidationFailedType failedType;
+ private int lineStart;
+ private int lineEnd;
+ private int errorLine;
+ private int errorColumn;
+ private String sql;
+ private String exception;
+
+ public Builder success(boolean success) {
+ this.success = success;
+ return this;
+ }
+
+ public Builder failedType(FlinkSqlValidationFailedType failedType) {
+ this.failedType = failedType;
+ return this;
+ }
+
+ public Builder lineStart(int lineStart) {
+ this.lineStart = lineStart;
+ return this;
+ }
+
+ public Builder lineEnd(int lineEnd) {
+ this.lineEnd = lineEnd;
+ return this;
+ }
+
+ public Builder errorLine(int errorLine) {
+ this.errorLine = errorLine;
+ return this;
+ }
+
+ public Builder errorColumn(int errorColumn) {
+ this.errorColumn = errorColumn;
+ return this;
+ }
+
+ public Builder sql(String sql) {
+ this.sql = sql;
+ return this;
+ }
+
+ public Builder exception(String exception) {
+ this.exception = exception;
+ return this;
+ }
+
+ public FlinkSqlValidationResult build() {
+ return new FlinkSqlValidationResult(this);
+ }
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidator.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidator.java
index ed6167433..c5fcec3dd 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidator.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkSqlValidator.java
@@ -19,6 +19,9 @@ package org.apache.streampark.flink.core;
import org.apache.streampark.common.enums.FlinkSqlValidationFailedType;
import org.apache.streampark.common.util.ExceptionUtils;
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
+
+import org.apache.streampark.shaded.org.slf4j.Logger;
import org.apache.calcite.config.Lex;
import org.apache.calcite.sql.parser.SqlParser;
@@ -27,144 +30,170 @@ import org.apache.flink.table.api.SqlDialect;
import org.apache.flink.table.api.config.TableConfigOptions;
import org.apache.flink.table.planner.delegation.FlinkSqlParserFactories;
-import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicReference;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-/** Validates Flink SQL syntax using Calcite parser. */
+/** Validates Flink SQL syntax. */
public final class FlinkSqlValidator {
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory()
+ .getLogger(FlinkSqlValidator.class.getName());
+
private static final String FLINK112_CALCITE_PARSER_CLASS =
"org.apache.flink.table.planner.calcite.CalciteParser";
private static final String FLINK113_PLUS_CALCITE_PARSER_CLASS =
"org.apache.flink.table.planner.parse.CalciteParser";
- private static final Pattern SYNTAX_ERROR_PATTERN =
+ private static final Pattern SYNTAX_ERROR_REGEXP =
Pattern.compile("at\\sline\\s(\\d+),\\scolumn\\s(\\d+)");
- private static final Map<String, SqlParser.Config> SQL_PARSER_CONFIG_MAP = createSqlParserConfigMap();
+ private static final Map<String, SqlParser.Config> SQL_PARSER_CONFIG_MAP;
- private static final Log LOG = new Log();
+ static {
+ Map<String, SqlParser.Config> configMap = new HashMap<>();
+ configMap.put(SqlDialect.DEFAULT.name(), getConfig(SqlDialect.DEFAULT));
+ configMap.put(SqlDialect.HIVE.name(), getConfig(SqlDialect.HIVE));
+ SQL_PARSER_CONFIG_MAP = Collections.unmodifiableMap(configMap);
+ }
private FlinkSqlValidator() {
}
public static FlinkSqlValidationResult verifySql(String sql) {
- AtomicReference<FlinkSqlValidationResult> earlyReturn = new AtomicReference<>();
+ final FlinkSqlValidationResult[] earlyReturn = new FlinkSqlValidationResult[1];
List<SqlCommandCall> sqlCommands =
- SqlCommandParser.parseSQL(sql, earlyReturn::set);
- FlinkSqlValidationResult earlyResult = earlyReturn.get();
- if (earlyResult != null) {
- return earlyResult;
+ SqlCommandParser.parseSQL(sql, result -> earlyReturn[0] = result);
+ if (earlyReturn[0] != null) {
+ return earlyReturn[0];
+ }
+ if (sqlCommands == null || sqlCommands.isEmpty()) {
+ return FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
+ .exception("verify failed: flink sql cannot be empty.")
+ .build();
}
- ValidationState state = new ValidationState();
+ ValidationContext context = new ValidationContext();
for (SqlCommandCall call : sqlCommands) {
- FlinkSqlValidationResult failure = processCommand(call, state);
- if (failure != null) {
- return failure;
+ FlinkSqlValidationResult validationError = validateCall(call, context);
+ if (validationError != null) {
+ return validationError;
}
}
- return state.toResult(sqlCommands);
+
+ if (context.hasInsert) {
+ return FlinkSqlValidationResult.ok();
+ }
+ return FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.SYNTAX_ERROR)
+ .lineStart(sqlCommands.get(0).lineStart)
+ .lineEnd(sqlCommands.get(sqlCommands.size() - 1).lineEnd)
+ .exception("No 'INSERT' statement to trigger the execution of the Flink job.")
+ .build();
}
- private static FlinkSqlValidationResult processCommand(SqlCommandCall call, ValidationState state) {
- SqlCommand command = call.command();
- switch (command) {
+ private static FlinkSqlValidationResult validateCall(SqlCommandCall call, ValidationContext context) {
+ switch (call.command) {
case SET:
+ context.updateDialect(call);
+ return null;
case RESET:
- state.updateDialect(command, call);
return null;
case BEGIN_STATEMENT_SET:
case END_STATEMENT_SET:
- LOG.warn("SQL Client Syntax: " + command.getCommandName());
+ LOG.warn("SQL Client Syntax: {} ", call.command.getName());
return null;
default:
- if (command == SqlCommand.INSERT) {
- state.markInsert();
+ if (call.command == SqlCommand.INSERT) {
+ context.hasInsert = true;
}
- try {
- validateSqlCommand(call, state.sqlDialect);
- } catch (IllegalStateException | UnsupportedOperationException e) {
- return syntaxErrorResult(call, e);
- }
- return null;
+ return parseWithCalcite(call, context.sqlDialect);
}
}
- private static void validateSqlCommand(SqlCommandCall call, String sqlDialect) {
- if ("HIVE".equalsIgnoreCase(sqlDialect)) {
- return;
- }
- if (!"DEFAULT".equalsIgnoreCase(sqlDialect)) {
- throw new UnsupportedOperationException("unsupported dialect: " + sqlDialect);
- }
+ private static FlinkSqlValidationResult parseWithCalcite(SqlCommandCall call, String sqlDialect) {
try {
- Class<?> calciteClass = resolveCalciteParserClass();
- Constructor<?> constructor = calciteClass.getConstructor(SqlParser.Config.class);
+ if ("HIVE".equalsIgnoreCase(sqlDialect)) {
+ return null;
+ }
+ if (!"DEFAULT".equalsIgnoreCase(sqlDialect)) {
+ throw new UnsupportedOperationException("unsupported dialect: " + sqlDialect);
+ }
+ Class<?> calciteClass = loadCalciteParserClass();
Object parser =
- constructor.newInstance(SQL_PARSER_CONFIG_MAP.get(sqlDialect.toUpperCase()));
+ calciteClass
+ .getConstructor(SqlParser.Config.class)
+ .newInstance(SQL_PARSER_CONFIG_MAP.get(sqlDialect.toUpperCase()));
Method method = parser.getClass().getDeclaredMethod("parse", String.class);
method.setAccessible(true);
- method.invoke(parser, call.originSql());
- } catch (ReflectiveOperationException e) {
- throw new IllegalStateException("Failed to parse SQL with Calcite", e);
- }
- }
-
- private static Class<?> resolveCalciteParserClass() throws ClassNotFoundException {
- try {
- return Class.forName(FLINK112_CALCITE_PARSER_CLASS);
- } catch (ClassNotFoundException e) {
- return Class.forName(FLINK113_PLUS_CALCITE_PARSER_CLASS);
+ method.invoke(parser, call.originSql);
+ return null;
+ } catch (Exception e) {
+ return toSyntaxErrorResult(call, e);
}
}
- private static FlinkSqlValidationResult syntaxErrorResult(SqlCommandCall call, Throwable e) {
+ private static FlinkSqlValidationResult toSyntaxErrorResult(SqlCommandCall call, Exception e) {
String exception = ExceptionUtils.stringifyException(e);
int causedByIndex = exception.indexOf("Caused by:");
- String causedBy = causedByIndex >= 0 ? exception.substring(causedByIndex) : exception;
- String cleanUpError = exception.replaceAll("[\r\n]", "");
- Matcher matcher = SYNTAX_ERROR_PATTERN.matcher(cleanUpError);
- if (matcher.find()) {
- int line = Integer.parseInt(matcher.group(1));
- int column = Integer.parseInt(matcher.group(2));
- int errorLine = call.lineStart() + line - 1;
- return FlinkSqlValidationResult.failure(
- FlinkSqlValidationFailedType.SYNTAX_ERROR,
- call.lineStart(),
- call.lineEnd(),
- errorLine,
- column,
- call.originSql(),
- causedBy.replaceAll("at\\sline\\s" + line, "at line " + errorLine));
+ String causedBy =
+ causedByIndex >= 0 ? exception.substring(causedByIndex) : exception;
+ Matcher syntaxMatcher = SYNTAX_ERROR_REGEXP.matcher(exception.replaceAll("[\r\n]", ""));
+ if (!syntaxMatcher.find()) {
+ return FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.SYNTAX_ERROR)
+ .lineStart(call.lineStart)
+ .lineEnd(call.lineEnd)
+ .sql(call.originSql)
+ .exception(causedBy)
+ .build();
}
- return FlinkSqlValidationResult.failure(
- FlinkSqlValidationFailedType.SYNTAX_ERROR,
- call.lineStart(),
- call.lineEnd(),
- 0,
- 0,
- call.originSql(),
- causedBy);
+ int line = Integer.parseInt(syntaxMatcher.group(1));
+ int column = Integer.parseInt(syntaxMatcher.group(2));
+ int errorLine = call.lineStart + line - 1;
+ return FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.SYNTAX_ERROR)
+ .lineStart(call.lineStart)
+ .lineEnd(call.lineEnd)
+ .errorLine(errorLine)
+ .errorColumn(column)
+ .sql(call.originSql)
+ .exception(causedBy.replaceAll("at\\sline\\s" + line, "at line " + errorLine))
+ .build();
}
- private static Map<String, SqlParser.Config> createSqlParserConfigMap() {
- Map<String, SqlParser.Config> map = new HashMap<>();
- map.put(SqlDialect.DEFAULT.name(), getConfig(SqlDialect.DEFAULT));
- map.put(SqlDialect.HIVE.name(), getConfig(SqlDialect.HIVE));
- return map;
+ private static Class<?> loadCalciteParserClass() throws ClassNotFoundException {
+ try {
+ return Class.forName(FLINK112_CALCITE_PARSER_CLASS);
+ } catch (ClassNotFoundException e) {
+ return Class.forName(FLINK113_PLUS_CALCITE_PARSER_CLASS);
+ }
}
private static SqlParser.Config getConfig(SqlDialect sqlDialect) {
- org.apache.calcite.sql.validate.SqlConformance conformance = FlinkSqlConformance.DEFAULT;
- if (sqlDialect != SqlDialect.DEFAULT && sqlDialect != SqlDialect.HIVE) {
+ FlinkSqlConformance conformance;
+ if (sqlDialect == SqlDialect.HIVE) {
+ try {
+ conformance = FlinkSqlConformance.DEFAULT;
+ } catch (NoSuchFieldError e) {
+ conformance = FlinkSqlConformance.DEFAULT;
+ } catch (Throwable e) {
+ throw new IllegalArgumentException("Init Flink sql Dialect error: ", e);
+ }
+ } else if (sqlDialect == SqlDialect.DEFAULT) {
+ conformance = FlinkSqlConformance.DEFAULT;
+ } else {
throw new UnsupportedOperationException("Unsupported sqlDialect: " + sqlDialect);
}
return SqlParser.config()
@@ -174,41 +203,19 @@ public final class FlinkSqlValidator {
.withIdentifierMaxLength(256);
}
- private static final class ValidationState {
+ private static final class ValidationContext {
private String sqlDialect = SqlDialect.DEFAULT.name().toLowerCase();
private boolean hasInsert;
- private void updateDialect(SqlCommand command, SqlCommandCall call) {
- if (command == SqlCommand.SET
- && call.operands()[0].equals(TableConfigOptions.TABLE_SQL_DIALECT.key())) {
- sqlDialect = call.operands()[call.operands().length - 1];
+ private void updateDialect(SqlCommandCall call) {
+ String args =
+ call.operands == null || call.operands.length == 0 ? null : call.operands[0];
+ if (args != null
+ && TableConfigOptions.TABLE_SQL_DIALECT.key().equals(args)
+ && call.operands.length > 1) {
+ sqlDialect = call.operands[call.operands.length - 1];
}
}
-
- private void markInsert() {
- hasInsert = true;
- }
-
- private FlinkSqlValidationResult toResult(List<SqlCommandCall> sqlCommands) {
- if (hasInsert) {
- return new FlinkSqlValidationResult();
- }
- return FlinkSqlValidationResult.failure(
- FlinkSqlValidationFailedType.SYNTAX_ERROR,
- sqlCommands.get(0).lineStart(),
- sqlCommands.get(sqlCommands.size() - 1).lineEnd(),
- 0,
- 0,
- null,
- "No 'INSERT' statement to trigger the execution of the Flink job.");
- }
- }
-
- private static final class Log extends org.apache.streampark.common.util.LoggerSupport {
-
- void warn(String msg) {
- logWarn(msg);
- }
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java
index d9006d515..852226621 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamTableTrait.java
@@ -38,13 +38,12 @@ import org.apache.flink.core.execution.JobListener;
import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.FileMonitoringFunction;
-import org.apache.flink.streaming.api.functions.source.FileProcessingMode;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.scala.DataStream;
-import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment;
import org.apache.flink.table.api.CompiledPlan;
import org.apache.flink.table.api.ExplainDetail;
import org.apache.flink.table.api.ExplainFormat;
@@ -53,8 +52,9 @@ import org.apache.flink.table.api.Schema;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableDescriptor;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.TableResult;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.catalog.Catalog;
import org.apache.flink.table.catalog.CatalogDescriptor;
import org.apache.flink.table.connector.ChangelogMode;
@@ -73,29 +73,31 @@ import org.apache.flink.util.SplittableIterator;
import com.esotericsoftware.kryo.Serializer;
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.Optional;
-
-import scala.Function1;
-import scala.Serializable;
-import scala.collection.Iterator;
-import scala.collection.Seq;
-import scala.runtime.BoxedUnit;
+import java.util.function.Consumer;
/**
- * Integration api of stream and table.
+ * Integration API of stream and table environments.
*
- * <p>Methods prefixed with {@code $} implement the Flink Scala {@code StreamTableEnvironment} API
- * and retain Scala-compatible names required by that interface.
+ * <p>Once a Table has been converted to a DataStream, the DataStream job must be executed using the
+ * execute method of the StreamExecutionEnvironment.
*/
+@SuppressWarnings("java:S100")
public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
public final ParameterTool parameter;
- public boolean isConvertedToDataStream = false;
private final StreamExecutionEnvironment streamEnv;
+
private final StreamTableEnvironment tableEnv;
+ /** Whether a table has been converted to a DataStream. */
+ public boolean isConvertedToDataStream;
+
protected FlinkStreamTableTrait(
ParameterTool parameter,
StreamExecutionEnvironment streamEnv,
@@ -105,46 +107,48 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
this.tableEnv = tableEnv;
}
- protected StreamExecutionEnvironment streamEnv() {
+ protected StreamExecutionEnvironment getStreamEnv() {
return streamEnv;
}
- protected StreamTableEnvironment tableEnv() {
+ protected StreamTableEnvironment getStreamTableEnv() {
return tableEnv;
}
- /**
- * Once a Table has been converted to a DataStream, the DataStream job must be executed using the
- * execute method of the StreamExecutionEnvironment.
- */
+ /** Recommended API to start tasks. */
public JobExecutionResult start() {
return start(null);
}
- /** Recommended to use this Api to start tasks */
public JobExecutionResult start(String name) {
- String appName = FlinkEnvironmentUtils.getAppName(parameter, name, true);
+ String appName = FlinkParameterUtils.getAppName(parameter, name, true);
return execute(appName);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
public JobExecutionResult execute(String jobName) {
Utils.printLogo("FlinkStreamTable " + jobName + " Starting...");
if (isConvertedToDataStream) {
- return streamEnv.execute(jobName);
+ try {
+ return streamEnv.execute(jobName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
}
return null;
}
- public void sql() {
- sql(null);
+ public void sql(String sql) {
+ sql(sql, null);
}
- public void sql(String sql) {
- FlinkSqlExecutor.executeSql(sql, parameter, this);
+ public void sql(String sql, Consumer<String> callback) {
+ FlinkSqlExecutor.executeSql(sql, parameter, this, callback);
}
- public org.apache.flink.streaming.api.environment.StreamExecutionEnvironment getJavaEnv() {
- return streamEnv.getJavaEnv();
+ public StreamExecutionEnvironment getJavaEnv() {
+ return streamEnv;
}
public List<Tuple2<String, DistributedCache.DistributedCacheEntry>> $getCachedFiles() {
@@ -200,7 +204,7 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
public CheckpointingMode $getCheckpointingMode() {
- return streamEnv.getCheckpointingMode();
+ return streamEnv.getCheckpointConfig().getCheckpointingMode();
}
public StreamExecutionEnvironment $setStateBackend(StateBackend backend) {
@@ -211,7 +215,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.getStateBackend();
}
- public void $setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) {
+ public void $setRestartStrategy(
+ RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) {
streamEnv.setRestartStrategy(restartStrategyConfiguration);
}
@@ -228,24 +233,22 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
public <T extends Serializer<?> & Serializable> void $addDefaultKryoSerializer(
- Class<?> type,
- T serializer) {
+ Class<?> type, T serializer) {
streamEnv.addDefaultKryoSerializer(type, serializer);
}
- public void $addDefaultKryoSerializer(Class<?> type, Class<? extends Serializer<?>> serializerClass) {
+ public void $addDefaultKryoSerializer(
+ Class<?> type, Class<? extends Serializer<?>> serializerClass) {
streamEnv.addDefaultKryoSerializer(type, serializerClass);
}
public <T extends Serializer<?> & Serializable> void $registerTypeWithKryoSerializer(
- Class<?> clazz,
- T serializer) {
+ Class<?> clazz, T serializer) {
streamEnv.registerTypeWithKryoSerializer(clazz, serializer);
}
public void $registerTypeWithKryoSerializer(
- Class<?> clazz,
- Class<? extends Serializer<?>> serializer) {
+ Class<?> clazz, Class<? extends Serializer<?>> serializer) {
streamEnv.registerTypeWithKryoSerializer(clazz, serializer);
}
@@ -261,25 +264,26 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
streamEnv.configure(configuration, classLoader);
}
- public DataStream<Object> $fromSequence(long from, long to) {
+ public DataStream<Long> $fromSequence(long from, long to) {
return streamEnv.fromSequence(from, to);
}
- public <T> DataStream<T> $fromElements(Seq<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromElements(data, typeInfo);
+ public <T> DataStream<T> $fromElements(T... data) {
+ return streamEnv.fromElements(data);
}
- public <T> DataStream<T> $fromCollection(Seq<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromCollection(data, typeInfo);
+ public <T> DataStream<T> $fromCollection(Collection<T> data) {
+ return streamEnv.fromCollection(data);
}
- public <T> DataStream<T> $fromCollection(Iterator<T> data, TypeInformation<T> typeInfo) {
- return streamEnv.fromCollection(data, typeInfo);
+ public <T> DataStream<T> $fromCollection(Iterator<T> data) {
+ java.util.List<T> list = new java.util.ArrayList<>();
+ data.forEachRemaining(list::add);
+ return streamEnv.fromCollection(list);
}
public <T> DataStream<T> $fromParallelCollection(
- SplittableIterator<T> data,
- TypeInformation<T> typeInfo) {
+ SplittableIterator<T> data, TypeInformation<T> typeInfo) {
return streamEnv.fromParallelCollection(data, typeInfo);
}
@@ -291,47 +295,36 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.readTextFile(filePath, charsetName);
}
- public <T> DataStream<T> $readFile(FileInputFormat<T> inputFormat, String filePath, TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, typeInfo);
+ public <T> DataStream<T> $readFile(FileInputFormat<T> inputFormat, String filePath) {
+ return streamEnv.readFile(inputFormat, filePath);
}
public <T> DataStream<T> $readFile(
FileInputFormat<T> inputFormat,
String filePath,
- FileProcessingMode watchType,
- long interval,
- TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, watchType, interval, typeInfo);
+ org.apache.flink.streaming.api.functions.source.FileProcessingMode watchType,
+ long interval) {
+ return streamEnv.readFile(inputFormat, filePath, watchType, interval);
}
public DataStream<String> $socketTextStream(
- String hostname,
- int port,
- char delimiter,
- long maxRetry) {
+ String hostname, int port, char delimiter, long maxRetry) {
return streamEnv.socketTextStream(hostname, port, delimiter, maxRetry);
}
- public <T> DataStream<T> $createInput(InputFormat<T, ?> inputFormat, TypeInformation<T> typeInfo) {
- return streamEnv.createInput(inputFormat, typeInfo);
- }
-
- public <T> DataStream<T> $addSource(SourceFunction<T> function, TypeInformation<T> typeInfo) {
- return streamEnv.addSource(function, typeInfo);
+ public <T> DataStream<T> $createInput(InputFormat<T, ?> inputFormat) {
+ return streamEnv.createInput(inputFormat);
}
- public <T> DataStream<T> $addSource(
- Function1<SourceFunction.SourceContext<T>, BoxedUnit> function,
- TypeInformation<T> typeInfo) {
- return streamEnv.addSource(function, typeInfo);
+ public <T> DataStream<T> $addSource(SourceFunction<T> function) {
+ return streamEnv.addSource(function);
}
public <T> DataStream<T> $fromSource(
Source<T, ? extends SourceSplit, ?> source,
WatermarkStrategy<T> watermarkStrategy,
- String sourceName,
- TypeInformation<T> typeInfo) {
- return streamEnv.fromSource(source, watermarkStrategy, sourceName, typeInfo);
+ String sourceName) {
+ return streamEnv.fromSource(source, watermarkStrategy, sourceName);
}
public void $registerJobListener(JobListener jobListener) {
@@ -342,11 +335,11 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
streamEnv.clearJobListeners();
}
- public JobClient $executeAsync() {
+ public JobClient $executeAsync() throws Exception {
return streamEnv.executeAsync();
}
- public JobClient $executeAsync(String jobName) {
+ public JobClient $executeAsync(String jobName) throws Exception {
return streamEnv.executeAsync(jobName);
}
@@ -358,8 +351,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.getStreamGraph();
}
- public org.apache.flink.streaming.api.environment.StreamExecutionEnvironment $getWrappedStreamExecutionEnvironment() {
- return streamEnv.getWrappedStreamExecutionEnvironment();
+ public StreamExecutionEnvironment $getWrappedStreamExecutionEnvironment() {
+ return streamEnv;
}
public void $registerCachedFile(String filePath, String name) {
@@ -371,28 +364,28 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
public boolean $isUnalignedCheckpointsEnabled() {
- return streamEnv.isUnalignedCheckpointsEnabled();
+ return streamEnv.getCheckpointConfig().isUnalignedCheckpointsEnabled();
}
public boolean $isForceUnalignedCheckpoints() {
- return streamEnv.isForceUnalignedCheckpoints();
+ return streamEnv.getCheckpointConfig().isForceUnalignedCheckpoints();
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
public StreamExecutionEnvironment $enableCheckpointing(
- long interval,
- CheckpointingMode mode,
- boolean force) {
- return streamEnv.enableCheckpointing(interval, mode, force);
- }
-
- public StreamExecutionEnvironment $enableCheckpointing() {
- return streamEnv.enableCheckpointing();
+ long interval, CheckpointingMode mode, boolean force) {
+ return streamEnv.enableCheckpointing(interval, mode);
}
- public DataStream<Object> $generateSequence(long from, long to) {
- return streamEnv.generateSequence(from, to);
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
+ public DataStream<Long> $generateSequence(long from, long to) {
+ return streamEnv.fromSequence(from, to);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
public DataStream<String> $readFileStream(
String streamPath,
long intervalMillis,
@@ -400,24 +393,15 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return streamEnv.readFileStream(streamPath, intervalMillis, watchType);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
public <T> DataStream<T> $readFile(
FileInputFormat<T> inputFormat,
String filePath,
- FileProcessingMode watchType,
+ org.apache.flink.streaming.api.functions.source.FileProcessingMode watchType,
long interval,
- FilePathFilter filter,
- TypeInformation<T> typeInfo) {
- return streamEnv.readFile(inputFormat, filePath, watchType, interval, filter, typeInfo);
- }
-
- @Override
- public <T> Table fromDataStream(DataStream<T> dataStream) {
- return tableEnv.fromDataStream(dataStream);
- }
-
- @Override
- public <T> Table fromDataStream(DataStream<T> dataStream, Seq<Expression> fields) {
- return tableEnv.fromDataStream(dataStream, fields);
+ FilePathFilter filter) {
+ return streamEnv.readFile(inputFormat, filePath, watchType, interval, filter);
}
@Override
@@ -436,20 +420,11 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public Table fromChangelogStream(DataStream<Row> dataStream, Schema schema, ChangelogMode changelogMode) {
+ public Table fromChangelogStream(
+ DataStream<Row> dataStream, Schema schema, ChangelogMode changelogMode) {
return tableEnv.fromChangelogStream(dataStream, schema, changelogMode);
}
- @Override
- public <T> void createTemporaryView(String path, DataStream<T> dataStream) {
- tableEnv.createTemporaryView(path, dataStream);
- }
-
- @Override
- public <T> void createTemporaryView(String path, DataStream<T> dataStream, Seq<Expression> fields) {
- tableEnv.createTemporaryView(path, dataStream, fields);
- }
-
@Override
public <T> void createTemporaryView(String path, DataStream<T> dataStream, Schema schema) {
tableEnv.createTemporaryView(path, dataStream, schema);
@@ -486,7 +461,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public DataStream<Row> toChangelogStream(Table table, Schema targetSchema, ChangelogMode changelogMode) {
+ public DataStream<Row> toChangelogStream(
+ Table table, Schema targetSchema, ChangelogMode changelogMode) {
isConvertedToDataStream = true;
return tableEnv.toChangelogStream(table, targetSchema, changelogMode);
}
@@ -498,11 +474,123 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public <T> DataStream<scala.Tuple2<Object, T>> toRetractStream(Table table, TypeInformation<T> typeInfo) {
+ public <T> DataStream<Tuple2<Boolean, T>> toRetractStream(Table table, Class<T> clazz) {
+ isConvertedToDataStream = true;
+ return tableEnv.toRetractStream(table, clazz);
+ }
+
+ @Override
+ public <T> DataStream<Tuple2<Boolean, T>> toRetractStream(
+ Table table, TypeInformation<T> typeInfo) {
isConvertedToDataStream = true;
return tableEnv.toRetractStream(table, typeInfo);
}
+ @Override
+ public void createCatalog(String catalogName, CatalogDescriptor catalogDescriptor) {
+ tableEnv.createCatalog(catalogName, catalogDescriptor);
+ }
+
+ @Override
+ public void useModules(String... moduleNames) {
+ tableEnv.useModules(moduleNames);
+ }
+
+ @Override
+ public void createFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createFunction(path, className, resourceUris);
+ }
+
+ @Override
+ public void createFunction(
+ String path,
+ String className,
+ List<ResourceUri> resourceUris,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
+ }
+
+ @Override
+ public void createTemporaryFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporaryFunction(path, className, resourceUris);
+ }
+
+ @Override
+ public void createTemporarySystemFunction(
+ String name, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporarySystemFunction(name, className, resourceUris);
+ }
+
+ @Override
+ public void createTemporaryTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTemporaryTable(path, descriptor);
+ }
+
+ @Override
+ public void createTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTable(path, descriptor);
+ }
+
+ @Override
+ public Table from(TableDescriptor descriptor) {
+ return tableEnv.from(descriptor);
+ }
+
+ @Override
+ public ModuleEntry[] listFullModules() {
+ return tableEnv.listFullModules();
+ }
+
+ @Override
+ public String[] listTables(String catalogName, String databaseName) {
+ return tableEnv.listTables(catalogName, databaseName);
+ }
+
+ @Override
+ public String explainSql(
+ String statement, ExplainFormat format, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, format, extraDetails);
+ }
+
+ @Override
+ public CompiledPlan loadPlan(PlanReference planReference) throws TableException {
+ return tableEnv.loadPlan(planReference);
+ }
+
+ @Override
+ public CompiledPlan compilePlanSql(String statement) throws TableException {
+ return tableEnv.compilePlanSql(statement);
+ }
+
+ @Override
+ public <T> Table fromDataStream(DataStream<T> dataStream) {
+ return tableEnv.fromDataStream(dataStream);
+ }
+
+ @Override
+ public <T> Table fromDataStream(DataStream<T> dataStream, Expression... fields) {
+ return tableEnv.fromDataStream(dataStream, fields);
+ }
+
+ @Override
+ public <T> void createTemporaryView(String path, DataStream<T> dataStream) {
+ tableEnv.createTemporaryView(path, dataStream);
+ }
+
+ @Override
+ public <T> void createTemporaryView(
+ String path, DataStream<T> dataStream, Expression... fields) {
+ tableEnv.createTemporaryView(path, dataStream, fields);
+ }
+
+ @Override
+ public <T> DataStream<T> toAppendStream(Table table, Class<T> clazz) {
+ isConvertedToDataStream = true;
+ return tableEnv.toAppendStream(table, clazz);
+ }
+
@Override
public Table fromValues(Expression... values) {
return tableEnv.fromValues(values);
@@ -545,13 +633,13 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
@Override
public void createTemporarySystemFunction(
- String name,
- Class<? extends UserDefinedFunction> functionClass) {
+ String name, Class<? extends UserDefinedFunction> functionClass) {
tableEnv.createTemporarySystemFunction(name, functionClass);
}
@Override
- public void createTemporarySystemFunction(String name, UserDefinedFunction functionInstance) {
+ public void createTemporarySystemFunction(
+ String name, UserDefinedFunction functionInstance) {
tableEnv.createTemporarySystemFunction(name, functionInstance);
}
@@ -579,7 +667,8 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
}
@Override
- public void createTemporaryFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
+ public void createTemporaryFunction(
+ String path, Class<? extends UserDefinedFunction> functionClass) {
tableEnv.createTemporaryFunction(path, functionClass);
}
@@ -698,127 +787,59 @@ public abstract class FlinkStreamTableTrait implements StreamTableEnvironment {
return tableEnv.getConfig();
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public <T> void registerFunction(String name, TableFunction<T> tf, TypeInformation<T> typeInfo) {
- tableEnv.registerFunction(name, tf, typeInfo);
+ public <T> void registerFunction(String name, TableFunction<T> function) {
+ tableEnv.registerFunction(name, function);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public <T, A> void registerFunction(
- String name,
- AggregateFunction<T, A> f,
- TypeInformation<T> typeInfo1,
- TypeInformation<A> typeInfo2) {
- tableEnv.registerFunction(name, f, typeInfo1, typeInfo2);
+ public <T, ACC> void registerFunction(String name, AggregateFunction<T, ACC> function) {
+ tableEnv.registerFunction(name, function);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public <T, A> void registerFunction(
- String name,
- TableAggregateFunction<T, A> f,
- TypeInformation<T> typeInfo1,
- TypeInformation<A> typeInfo2) {
- tableEnv.registerFunction(name, f, typeInfo1, typeInfo2);
+ public <T, ACC> void registerFunction(String name, TableAggregateFunction<T, ACC> function) {
+ tableEnv.registerFunction(name, function);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
public <T> void registerDataStream(String name, DataStream<T> dataStream) {
tableEnv.registerDataStream(name, dataStream);
}
- @Override
- public <T> void registerDataStream(String name, DataStream<T> dataStream, Seq<Expression> fields) {
- tableEnv.registerDataStream(name, dataStream, fields);
- }
-
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
public void registerFunction(String name, ScalarFunction function) {
tableEnv.registerFunction(name, function);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
public void registerTable(String name, Table table) {
tableEnv.registerTable(name, table);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
public Table scan(String... tablePath) {
return tableEnv.scan(tablePath);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
public String[] getCompletionHints(String statement, int position) {
return tableEnv.getCompletionHints(statement, position);
}
-
- @Override
- public void useModules(String... modules) {
- tableEnv.useModules(modules);
- }
-
- @Override
- public ModuleEntry[] listFullModules() {
- return tableEnv.listFullModules();
- }
-
- @Override
- public void createTemporaryTable(String path, TableDescriptor descriptor) {
- tableEnv.createTemporaryTable(path, descriptor);
- }
-
- @Override
- public void createTable(String path, TableDescriptor descriptor) {
- tableEnv.createTable(path, descriptor);
- }
-
- @Override
- public Table from(TableDescriptor descriptor) {
- return tableEnv.from(descriptor);
- }
-
- @Override
- public String[] listTables(String catalogName, String databaseName) {
- return tableEnv.listTables(catalogName, databaseName);
- }
-
- @Override
- public CompiledPlan loadPlan(PlanReference planReference) {
- return tableEnv.loadPlan(planReference);
- }
-
- @Override
- public CompiledPlan compilePlanSql(String stmt) {
- return tableEnv.compilePlanSql(stmt);
- }
-
- @Override
- public void createFunction(String path, String className, List<ResourceUri> resourceUris) {
- tableEnv.createFunction(path, className, resourceUris);
- }
-
- @Override
- public void createFunction(
- String path, String className, List<ResourceUri> resourceUris, boolean ignoreIfExists) {
- tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
- }
-
- @Override
- public void createTemporaryFunction(String path, String className, List<ResourceUri> resourceUris) {
- tableEnv.createTemporaryFunction(path, className, resourceUris);
- }
-
- @Override
- public void createTemporarySystemFunction(String name, String className, List<ResourceUri> resourceUris) {
- tableEnv.createTemporarySystemFunction(name, className, resourceUris);
- }
-
- @Override
- public String explainSql(String statement, ExplainFormat format, ExplainDetail... extraDetails) {
- return tableEnv.explainSql(statement, format, extraDetails);
- }
-
- @Override
- public void createCatalog(String catalog, CatalogDescriptor catalogDescriptor) {
- tableEnv.createCatalog(catalog, catalogDescriptor);
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializer.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializer.java
index 94f872232..8f232989f 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializer.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkStreamingInitializer.java
@@ -18,7 +18,6 @@
package org.apache.streampark.flink.core;
import org.apache.streampark.common.conf.ConfigKeys;
-import org.apache.streampark.common.enums.ApiType;
import org.apache.streampark.common.util.DeflaterUtils;
import org.apache.streampark.common.util.FileUtils;
import org.apache.streampark.common.util.HdfsUtils;
@@ -27,33 +26,44 @@ import org.apache.streampark.flink.core.conf.FlinkConfiguration;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
-import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import java.io.File;
+import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
-/** Initializes Flink streaming execution environment from CLI args and config files. */
-public class FlinkStreamingInitializer extends org.apache.streampark.common.util.LoggerSupport {
+/** Initializes Flink streaming execution environment from application arguments. */
+public class FlinkStreamingInitializer {
- protected final String[] args;
- protected final ApiType apiType;
+ final String[] args;
- scala.Function2<org.apache.flink.streaming.api.scala.StreamExecutionEnvironment, ParameterTool, scala.runtime.BoxedUnit> streamEnvConfFunc;
- scala.Function2<TableConfig, ParameterTool, scala.runtime.BoxedUnit> tableConfFunc;
StreamEnvConfigFunction javaStreamEnvConfFunc;
- TableEnvConfigFunction javaTableEnvConfFunc;
private FlinkConfiguration configuration;
- private org.apache.flink.streaming.api.scala.StreamExecutionEnvironment streamEnv;
- protected FlinkStreamingInitializer(String[] args, ApiType apiType) {
+ private StreamExecutionEnvironment streamEnv;
+
+ FlinkStreamingInitializer(String[] args) {
this.args = args;
- this.apiType = apiType;
}
- ParameterTool parameter() {
- return getConfiguration().parameter();
+ public static StreamingInitResult initialize(String[] args, StreamEnvConfigFunction config) {
+ FlinkStreamingInitializer flinkInitializer = new FlinkStreamingInitializer(args);
+ flinkInitializer.javaStreamEnvConfFunc = config;
+ return new StreamingInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getStreamEnv());
+ }
+
+ public static StreamingInitResult initialize(StreamEnvConfig args) {
+ FlinkStreamingInitializer flinkInitializer = new FlinkStreamingInitializer(args.args);
+ flinkInitializer.javaStreamEnvConfFunc = args.conf;
+ return new StreamingInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getStreamEnv());
+ }
+
+ ParameterTool getParameter() {
+ return getConfiguration().parameter;
}
FlinkConfiguration getConfiguration() {
@@ -63,43 +73,31 @@ public class FlinkStreamingInitializer extends org.apache.streampark.common.util
return configuration;
}
- org.apache.flink.streaming.api.scala.StreamExecutionEnvironment getStreamEnv() {
+ StreamExecutionEnvironment getStreamEnv() {
if (streamEnv == null) {
- org.apache.flink.streaming.api.scala.StreamExecutionEnvironment env =
- new org.apache.flink.streaming.api.scala.StreamExecutionEnvironment(
- org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getExecutionEnvironment(
- getConfiguration().envConfig()));
- switch (apiType) {
- case JAVA:
- if (javaStreamEnvConfFunc != null) {
- javaStreamEnvConfFunc.configuration(env.getJavaEnv(), getConfiguration().parameter());
- }
- break;
- case SCALA:
- if (streamEnvConfFunc != null) {
- streamEnvConfFunc.apply(env, getConfiguration().parameter());
- }
- break;
- default:
- break;
+ streamEnv =
+ StreamExecutionEnvironment.getExecutionEnvironment(
+ getConfiguration().envConfig);
+ if (javaStreamEnvConfFunc != null) {
+ javaStreamEnvConfFunc.configuration(streamEnv, getConfiguration().parameter);
}
- env.getConfig().setGlobalJobParameters(getConfiguration().parameter());
- streamEnv = env;
+ streamEnv.getConfig().setGlobalJobParameters(getConfiguration().parameter);
}
return streamEnv;
}
FlinkConfiguration initParameter() {
ParameterTool argsMap = ParameterTool.fromArgs(args);
- String configPath = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
- if (configPath == null || configPath.isEmpty()) {
+ String configFile = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
+ if (configFile == null || configFile.isEmpty()) {
throw new ExceptionInInitializerError(
"[StreamPark] Usage:can't find config,please set \"--conf $path \" in main arguments");
}
- Map<String, String> configMap = parseConfig(configPath);
+ Map<String, String> configMap = parseConfig(configFile);
Map<String, String> properConf =
extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_PROPERTY_PREFIX());
- Map<String, String> appConf = extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
+ Map<String, String> appConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
ParameterTool parameter =
ParameterTool.fromSystemProperties()
@@ -112,49 +110,50 @@ public class FlinkStreamingInitializer extends org.apache.streampark.common.util
}
Map<String, String> parseConfig(String config) {
- String format = config.contains(".")
- ? config.substring(config.lastIndexOf('.') + 1).toLowerCase()
- : "";
- Map<String, String> map = readConfigContent(config, format);
- Map<String, String> filtered = new HashMap<>();
- map.forEach((key, value) -> {
- if (value != null && !value.isEmpty()) {
- filtered.put(key, value);
- }
- });
- return filtered;
- }
-
- private Map<String, String> readConfigContent(String config, String format) {
+ Map<String, String> map;
if (config.startsWith("yaml://")) {
- return readConfigText(format, DeflaterUtils.unzipString(config.substring(7)));
- }
- if (config.startsWith("conf://")) {
- return readConfigText(format, DeflaterUtils.unzipString(config.substring(7)));
- }
- if (config.startsWith("prop://")) {
- return readConfigText(format, DeflaterUtils.unzipString(config.substring(7)));
- }
- if (config.startsWith("hdfs://")) {
+ map = PropertiesUtils.fromYamlText(DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("conf://")) {
+ map = PropertiesUtils.fromHoconText(DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("prop://")) {
+ map =
+ PropertiesUtils.fromPropertiesText(
+ DeflaterUtils.unzipString(config.substring(7)));
+ } else if (config.startsWith("hdfs://")) {
try {
- return readConfigText(format, HdfsUtils.read(config));
- } catch (java.io.IOException e) {
- throw new IllegalStateException("Failed to read HDFS config: " + config, e);
+ String text = HdfsUtils.read(config);
+ map = readConfig(config, text);
+ } catch (IOException e) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Failed to read application config from HDFS: " + config, e);
+ }
+ } else {
+ File file = new File(config);
+ if (!file.exists()) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Usage: application config file: "
+ + file
+ + " is not found!!!");
+ }
+ try {
+ map = readConfig(config, FileUtils.readFile(file));
+ } catch (IOException e) {
+ throw new IllegalArgumentException(
+ "[StreamPark] Failed to read application config file: " + config, e);
}
}
- File configFile = new File(config);
- if (!configFile.exists()) {
- throw new IllegalArgumentException(
- "[StreamPark] Usage: application config file: " + configFile + " is not found!!!");
- }
- try {
- return readConfigText(format, FileUtils.readFile(configFile));
- } catch (java.io.IOException e) {
- throw new IllegalStateException("Failed to read config file: " + configFile, e);
- }
+ Map<String, String> filtered = new HashMap<>();
+ map.forEach(
+ (key, value) -> {
+ if (value != null && !value.isEmpty()) {
+ filtered.put(key, value);
+ }
+ });
+ return filtered;
}
- private Map<String, String> readConfigText(String format, String text) {
+ private Map<String, String> readConfig(String config, String text) {
+ String format = config.substring(config.lastIndexOf('.') + 1).toLowerCase();
switch (format) {
case "yml":
case "yaml":
@@ -170,13 +169,25 @@ public class FlinkStreamingInitializer extends org.apache.streampark.common.util
}
Map<String, String> extractConfigByPrefix(Map<String, String> configMap, String prefix) {
- Map<String, String> result = new HashMap<>();
+ Map<String, String> map = new HashMap<>();
configMap.forEach(
(key, value) -> {
if (key.startsWith(prefix)) {
- result.put(key.substring(prefix.length()), value);
+ map.put(key.substring(prefix.length()), value);
}
});
- return result;
+ return map;
+ }
+
+ /** Streaming initialization result. */
+ public static final class StreamingInitResult {
+
+ public final ParameterTool parameter;
+ public final StreamExecutionEnvironment streamEnv;
+
+ StreamingInitResult(ParameterTool parameter, StreamExecutionEnvironment streamEnv) {
+ this.parameter = parameter;
+ this.streamEnv = streamEnv;
+ }
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializer.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializer.java
index 0c59627cb..34c1e6b94 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializer.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableInitializer.java
@@ -18,145 +18,131 @@
package org.apache.streampark.flink.core;
import org.apache.streampark.common.conf.ConfigKeys;
-import org.apache.streampark.common.enums.ApiType;
import org.apache.streampark.common.enums.PlannerType;
import org.apache.streampark.common.util.DeflaterUtils;
import org.apache.streampark.common.util.PropertiesUtils;
+import org.apache.streampark.common.util.StreamParkLoggerFactory;
import org.apache.streampark.flink.core.conf.FlinkConfiguration;
+import org.apache.streampark.shaded.org.slf4j.Logger;
+
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import java.io.File;
import java.lang.reflect.Method;
import java.util.HashMap;
import java.util.Map;
-import scala.Tuple2;
-import scala.Tuple3;
-
-/** Initializes Flink Table / StreamTable environments from CLI args and config files. */
+/** Initializes Flink table and stream-table environments from application arguments. */
public class FlinkTableInitializer extends FlinkStreamingInitializer {
+ private static final Logger LOG =
+ StreamParkLoggerFactory.loggerFactory()
+ .getLogger(FlinkTableInitializer.class.getName());
+
+ private TableEnvConfigFunction javaTableEnvConfFunc;
+
private EnvironmentSettings.Builder envSettingsBuilder;
- private TableEnvironment tableEnv;
- private StreamTableEnvironment streamTableEnv;
- private FlinkTableInitializer(String[] args, ApiType apiType) {
- super(args, apiType);
- }
+ private TableEnvironment tableEnv;
- public static Tuple2<ParameterTool, TableEnvironment> initialize(
- String[] args,
- scala.Function2<TableConfig, ParameterTool, scala.runtime.BoxedUnit> config) {
- FlinkTableInitializer initializer = new FlinkTableInitializer(args, ApiType.SCALA);
- initializer.tableConfFunc = config;
- return new Tuple2<>(
- initializer.getConfiguration().parameter(),
- initializer.getTableEnv());
- }
+ private StreamTableEnvironment streamTableEnv;
- public static Tuple2<ParameterTool, TableEnvironment> initialize(TableEnvConfig args) {
- FlinkTableInitializer initializer = new FlinkTableInitializer(args.args(), ApiType.JAVA);
- initializer.javaTableEnvConfFunc = args.conf();
- return new Tuple2<>(
- initializer.getConfiguration().parameter(),
- initializer.getTableEnv());
+ FlinkTableInitializer(String[] args) {
+ super(args);
}
- public static Tuple3<ParameterTool, org.apache.flink.streaming.api.scala.StreamExecutionEnvironment, StreamTableEnvironment> initialize(
- String[] args,
- scala.Function2<org.apache.flink.streaming.api.scala.StreamExecutionEnvironment, ParameterTool, scala.runtime.BoxedUnit> configStream,
- scala.Function2<TableConfig, ParameterTool, scala.runtime.BoxedUnit> configTable) {
- FlinkTableInitializer initializer = new FlinkTableInitializer(args, ApiType.SCALA);
- initializer.streamEnvConfFunc = configStream;
- initializer.tableConfFunc = configTable;
- return new Tuple3<>(
- initializer.getConfiguration().parameter(),
- initializer.getStreamEnv(),
- initializer.getStreamTableEnv());
+ public static TableInitResult initialize(TableEnvConfig args) {
+ FlinkTableInitializer flinkInitializer = new FlinkTableInitializer(args.args);
+ flinkInitializer.javaTableEnvConfFunc = args.conf;
+ return new TableInitResult(
+ flinkInitializer.getConfiguration().parameter, flinkInitializer.getTableEnv());
}
- public static Tuple3<ParameterTool, org.apache.flink.streaming.api.scala.StreamExecutionEnvironment, StreamTableEnvironment> initialize(
- StreamTableEnvConfig args) {
- FlinkTableInitializer initializer = new FlinkTableInitializer(args.args(), ApiType.JAVA);
- initializer.javaStreamEnvConfFunc = args.streamConfig();
- initializer.javaTableEnvConfFunc = args.tableConfig();
- return new Tuple3<>(
- initializer.getConfiguration().parameter(),
- initializer.getStreamEnv(),
- initializer.getStreamTableEnv());
+ public static StreamTableInitResult initialize(StreamTableEnvConfig args) {
+ FlinkTableInitializer flinkInitializer = new FlinkTableInitializer(args.args);
+ flinkInitializer.javaStreamEnvConfFunc = args.streamConfig;
+ flinkInitializer.javaTableEnvConfFunc = args.tableConfig;
+ return new StreamTableInitResult(
+ flinkInitializer.getConfiguration().parameter,
+ flinkInitializer.getStreamEnv(),
+ flinkInitializer.getStreamTableEnv());
}
- private EnvironmentSettings.Builder getEnvSettingsBuilder() {
- if (envSettingsBuilder == null) {
- envSettingsBuilder = buildEnvSettings();
- }
- return envSettingsBuilder;
+ public static StreamTableInitResult initialize(
+ String[] args,
+ StreamEnvConfigFunction streamConfig,
+ TableEnvConfigFunction tableConfig) {
+ FlinkTableInitializer flinkInitializer = new FlinkTableInitializer(args);
+ flinkInitializer.javaStreamEnvConfFunc = streamConfig;
+ flinkInitializer.javaTableEnvConfFunc = tableConfig;
+ return new StreamTableInitResult(
+ flinkInitializer.getConfiguration().parameter,
+ flinkInitializer.getStreamEnv(),
+ flinkInitializer.getStreamTableEnv());
}
TableEnvironment getTableEnv() {
if (tableEnv == null) {
- logInfo("job working in batch mode");
+ LOG.info("job working in batch mode");
EnvironmentSettings.Builder builder = getEnvSettingsBuilder();
builder.inBatchMode();
tableEnv =
- FlinkEnvironmentUtils.setAppName(
- TableEnvironment.create(builder.build()), parameter());
- applyTableConfig(tableEnv.getConfig());
+ FlinkParameterUtils.setAppName(
+ TableEnvironment.create(builder.build()), getParameter());
+ applyTableEnvConfig(tableEnv.getConfig());
}
return tableEnv;
}
StreamTableEnvironment getStreamTableEnv() {
if (streamTableEnv == null) {
- logInfo("components should work in streaming mode");
+ LOG.info("components should work in streaming mode");
EnvironmentSettings.Builder builder = getEnvSettingsBuilder();
builder.inStreamingMode();
EnvironmentSettings setting = builder.build();
- if (streamEnvConfFunc != null) {
- streamEnvConfFunc.apply(getStreamEnv(), parameter());
- }
if (javaStreamEnvConfFunc != null) {
- javaStreamEnvConfFunc.configuration(getStreamEnv().getJavaEnv(), parameter());
+ javaStreamEnvConfFunc.configuration(getStreamEnv(), getParameter());
}
streamTableEnv =
- FlinkEnvironmentUtils.setAppName(
- StreamTableEnvironment.create(getStreamEnv(), setting), parameter());
- applyTableConfig(streamTableEnv.getConfig());
+ FlinkParameterUtils.setAppName(
+ StreamTableEnvironment.create(getStreamEnv(), setting), getParameter());
+ applyTableEnvConfig(streamTableEnv.getConfig());
}
return streamTableEnv;
}
- private void applyTableConfig(TableConfig config) {
- switch (apiType()) {
- case JAVA:
- if (javaTableEnvConfFunc != null) {
- javaTableEnvConfFunc.configuration(config, parameter());
- }
- break;
- case SCALA:
- if (tableConfFunc != null) {
- tableConfFunc.apply(config, parameter());
- }
- break;
- default:
- break;
+ private void applyTableEnvConfig(TableConfig config) {
+ if (javaTableEnvConfFunc != null) {
+ javaTableEnvConfFunc.configuration(config, getParameter());
}
}
- private EnvironmentSettings.Builder buildEnvSettings() {
+ private EnvironmentSettings.Builder getEnvSettingsBuilder() {
+ if (envSettingsBuilder == null) {
+ envSettingsBuilder = buildEnvSettings(getParameter());
+ }
+ return envSettingsBuilder;
+ }
+
+ private EnvironmentSettings.Builder buildEnvSettings(ParameterTool parameter) {
EnvironmentSettings.Builder builder = EnvironmentSettings.newInstance();
- PlannerType plannerType;
- try {
- plannerType = PlannerType.withName(parameter().get(ConfigKeys.KEY_FLINK_TABLE_PLANNER()));
- } catch (Exception e) {
- plannerType = PlannerType.BLINK;
+
+ PlannerType plannerType = PlannerType.BLINK;
+ String plannerName = parameter.get(ConfigKeys.KEY_FLINK_TABLE_PLANNER(), null);
+ if (plannerName != null && !plannerName.isEmpty()) {
+ try {
+ plannerType = PlannerType.withName(plannerName);
+ } catch (IllegalArgumentException e) {
+ plannerType = PlannerType.BLINK;
+ }
}
switch (plannerType) {
@@ -173,7 +159,7 @@ public class FlinkTableInitializer extends FlinkStreamingInitializer {
break;
}
- String flinkConf = parameter().get(ConfigKeys.KEY_FLINK_CONF(), null);
+ String flinkConf = parameter.get(ConfigKeys.KEY_FLINK_CONF(), null);
if (flinkConf == null || flinkConf.isEmpty()) {
throw new ExceptionInInitializerError(
"[StreamPark] Usage:can't find config,please set \"--flink.conf $conf \" in main arguments");
@@ -182,53 +168,51 @@ public class FlinkTableInitializer extends FlinkStreamingInitializer {
Configuration.fromMap(
PropertiesUtils.fromYamlText(DeflaterUtils.unzipString(flinkConf))));
- String catalog = parameter().get(ConfigKeys.KEY_FLINK_TABLE_CATALOG(), null);
- String database = parameter().get(ConfigKeys.KEY_FLINK_TABLE_DATABASE(), null);
+ String catalog = parameter.get(ConfigKeys.KEY_FLINK_TABLE_CATALOG(), null);
+ String database = parameter.get(ConfigKeys.KEY_FLINK_TABLE_DATABASE(), null);
if (catalog != null && database != null) {
- logInfo("with built in catalog: " + catalog);
- logInfo("with built in database: " + database);
+ LOG.info("with built in catalog: {}", catalog);
+ LOG.info("with built in database: {}", database);
builder.withBuiltInCatalogName(catalog);
builder.withBuiltInDatabaseName(database);
} else if (catalog != null) {
- logInfo("with built in catalog: " + catalog);
+ LOG.info("with built in catalog: {}", catalog);
builder.withBuiltInCatalogName(catalog);
} else if (database != null) {
- logInfo("with built in database: " + database);
+ LOG.info("with built in database: {}", database);
builder.withBuiltInDatabaseName(database);
}
return builder;
}
- private boolean invokePlannerMethod(
- EnvironmentSettings.Builder builder, String methodName, String successMessage) {
+ private void invokePlannerMethod(
+ EnvironmentSettings.Builder builder, String methodName, String successMessage) {
try {
Method method = builder.getClass().getDeclaredMethod(methodName);
method.setAccessible(true);
method.invoke(builder);
if (successMessage != null) {
- logInfo(successMessage);
+ LOG.info(successMessage);
}
- return true;
} catch (NoSuchMethodException e) {
- logWarn(methodName + " deprecated");
- return false;
+ LOG.warn("{} deprecated", methodName);
} catch (ReflectiveOperationException e) {
- logWarn(methodName + " deprecated");
- return false;
+ LOG.warn("Failed to invoke {} on EnvironmentSettings.Builder", methodName, e);
}
}
@Override
FlinkConfiguration initParameter() {
- ParameterTool argsMap = ParameterTool.fromArgs(args());
- String configPath = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
+ ParameterTool argsMap = ParameterTool.fromArgs(args);
+ String configFile = argsMap.get(ConfigKeys.KEY_APP_CONF(), null);
FlinkConfiguration configuration;
- if (configPath == null || configPath.isEmpty()) {
- logWarn("Usage:can't find config,you can set \"--conf $path \" in main arguments");
+ if (configFile == null || configFile.isEmpty()) {
+ LOG.warn("Usage:can't find config,you can set \"--conf $path \" in main arguments");
ParameterTool parameter = ParameterTool.fromSystemProperties().mergeWith(argsMap);
- configuration = new FlinkConfiguration(parameter, new Configuration(), new Configuration());
+ configuration =
+ new FlinkConfiguration(parameter, new Configuration(), new Configuration());
} else {
- Map<String, String> configMap = parseConfig(configPath);
+ Map<String, String> configMap = parseConfig(configFile);
Map<String, String> sqlConf = new HashMap<>();
configMap.forEach(
(key, value) -> {
@@ -239,11 +223,12 @@ public class FlinkTableInitializer extends FlinkStreamingInitializer {
Map<String, String> properConf =
extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_PROPERTY_PREFIX());
- Map<String, String> appConf = extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
+ Map<String, String> appConf =
+ extractConfigByPrefix(configMap, ConfigKeys.KEY_APP_PREFIX());
Map<String, String> tableConf =
extractConfigByPrefix(configMap, ConfigKeys.KEY_FLINK_TABLE_PREFIX());
- Configuration tableConfiguration = Configuration.fromMap(tableConf);
+ Configuration tableConfig = Configuration.fromMap(tableConf);
Configuration envConfig = Configuration.fromMap(properConf);
ParameterTool parameter =
@@ -254,37 +239,59 @@ public class FlinkTableInitializer extends FlinkStreamingInitializer {
.mergeWith(ParameterTool.fromMap(sqlConf))
.mergeWith(argsMap);
- configuration = new FlinkConfiguration(parameter, envConfig, tableConfiguration);
+ configuration = new FlinkConfiguration(parameter, envConfig, tableConfig);
}
- String flinkSql = configuration.parameter().get(ConfigKeys.KEY_FLINK_SQL(), null);
+ String flinkSql = configuration.parameter.get(ConfigKeys.KEY_FLINK_SQL(), null);
if (flinkSql == null) {
return configuration;
}
+
try {
String value = DeflaterUtils.unzipString(flinkSql);
- Map<String, String> sqlMap = new HashMap<>();
- sqlMap.put(ConfigKeys.KEY_FLINK_SQL(), value);
return configuration.withParameter(
- configuration.parameter().mergeWith(ParameterTool.fromMap(sqlMap)));
+ configuration.parameter.mergeWith(
+ ParameterTool.fromMap(
+ Map.of(ConfigKeys.KEY_FLINK_SQL(), value))));
} catch (Exception ignored) {
File sqlFile = new File(flinkSql);
try {
Map<String, String> value =
PropertiesUtils.fromYamlFile(sqlFile.getAbsolutePath());
return configuration.withParameter(
- configuration.parameter().mergeWith(ParameterTool.fromMap(value)));
+ configuration.parameter.mergeWith(ParameterTool.fromMap(value)));
} catch (Exception e) {
- throw new IllegalArgumentException("[StreamPark] init sql error." + e, e);
+ throw new IllegalArgumentException("[StreamPark] init sql error." + e);
}
}
}
- private String[] args() {
- return args;
+ /** Table initialization result. */
+ public static final class TableInitResult {
+
+ public final ParameterTool parameter;
+ public final TableEnvironment tableEnv;
+
+ public TableInitResult(ParameterTool parameter, TableEnvironment tableEnv) {
+ this.parameter = parameter;
+ this.tableEnv = tableEnv;
+ }
}
- private ApiType apiType() {
- return apiType;
+ /** Stream-table initialization result. */
+ public static final class StreamTableInitResult {
+
+ public final ParameterTool parameter;
+ public final StreamExecutionEnvironment streamEnv;
+ public final StreamTableEnvironment streamTableEnv;
+
+ public StreamTableInitResult(
+ ParameterTool parameter,
+ StreamExecutionEnvironment streamEnv,
+ StreamTableEnvironment streamTableEnv) {
+ this.parameter = parameter;
+ this.streamEnv = streamEnv;
+ this.streamTableEnv = streamTableEnv;
+ }
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
index c8d47444b..aa8d2152e 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/FlinkTableTrait.java
@@ -30,6 +30,7 @@ import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableDescriptor;
import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.TableResult;
import org.apache.flink.table.catalog.Catalog;
import org.apache.flink.table.catalog.CatalogDescriptor;
@@ -44,9 +45,11 @@ import org.apache.flink.table.types.AbstractDataType;
import java.util.List;
import java.util.Optional;
+/** Base table environment trait with SQL execution helpers. */
public abstract class FlinkTableTrait implements TableEnvironment {
public final ParameterTool parameter;
+
private final TableEnvironment tableEnv;
protected FlinkTableTrait(ParameterTool parameter, TableEnvironment tableEnv) {
@@ -54,315 +57,328 @@ public abstract class FlinkTableTrait implements TableEnvironment {
this.tableEnv = tableEnv;
}
- protected TableEnvironment delegate() {
+ protected TableEnvironment getTableEnv() {
return tableEnv;
}
public JobExecutionResult start() {
- String appName = FlinkEnvironmentUtils.getAppName(parameter, null, true);
- return printStartupLogo(appName);
+ String appName = FlinkParameterUtils.getAppName(parameter, true);
+ return execute(appName);
}
- JobExecutionResult printStartupLogo(String jobName) {
+ public JobExecutionResult execute(String jobName) {
Utils.printLogo("FlinkTable " + jobName + " Starting...");
return null;
}
- public void sql() {
- sql(null);
- }
-
public void sql(String sql) {
FlinkSqlExecutor.executeSql(sql, parameter, this);
}
@Override
public Table fromValues(Expression... values) {
- return delegate().fromValues(values);
+ return tableEnv.fromValues(values);
}
@Override
public Table fromValues(AbstractDataType<?> rowType, Expression... values) {
- return delegate().fromValues(rowType, values);
+ return tableEnv.fromValues(rowType, values);
}
@Override
public Table fromValues(Iterable<?> values) {
- return delegate().fromValues(values);
+ return tableEnv.fromValues(values);
}
@Override
public Table fromValues(AbstractDataType<?> rowType, Iterable<?> values) {
- return delegate().fromValues(rowType, values);
+ return tableEnv.fromValues(rowType, values);
}
@Override
- public void registerCatalog(String catalogName, Catalog catalog) {
- delegate().registerCatalog(catalogName, catalog);
+ public void createCatalog(String catalogName, CatalogDescriptor catalogDescriptor) {
+ tableEnv.createCatalog(catalogName, catalogDescriptor);
}
@Override
- public Optional<Catalog> getCatalog(String catalogName) {
- return delegate().getCatalog(catalogName);
+ public void useModules(String... moduleNames) {
+ tableEnv.useModules(moduleNames);
}
@Override
- public void loadModule(String moduleName, Module module) {
- delegate().loadModule(moduleName, module);
+ public void createFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createFunction(path, className, resourceUris);
}
@Override
- public void unloadModule(String moduleName) {
- delegate().unloadModule(moduleName);
+ public void createFunction(
+ String path,
+ String className,
+ List<ResourceUri> resourceUris,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, className, resourceUris, ignoreIfExists);
}
@Override
- public void createTemporarySystemFunction(
- String name, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createTemporarySystemFunction(name, functionClass);
+ public void createTemporaryFunction(
+ String path, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporaryFunction(path, className, resourceUris);
}
@Override
- public void createTemporarySystemFunction(String name, UserDefinedFunction functionInstance) {
- delegate().createTemporarySystemFunction(name, functionInstance);
+ public void createTemporarySystemFunction(
+ String name, String className, List<ResourceUri> resourceUris) {
+ tableEnv.createTemporarySystemFunction(name, className, resourceUris);
}
@Override
- public boolean dropTemporarySystemFunction(String name) {
- return delegate().dropTemporarySystemFunction(name);
+ public void createTemporaryTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTemporaryTable(path, descriptor);
}
@Override
- public void createFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createFunction(path, functionClass);
+ public void createTable(String path, TableDescriptor descriptor) {
+ tableEnv.createTable(path, descriptor);
}
@Override
- public void createFunction(
- String path,
- Class<? extends UserDefinedFunction> functionClass,
- boolean ignoreIfExists) {
- delegate().createFunction(path, functionClass, ignoreIfExists);
+ public Table from(TableDescriptor descriptor) {
+ return tableEnv.from(descriptor);
}
@Override
- public boolean dropFunction(String path) {
- return delegate().dropFunction(path);
+ public ModuleEntry[] listFullModules() {
+ return tableEnv.listFullModules();
}
@Override
- public void createTemporaryFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
- delegate().createTemporaryFunction(path, functionClass);
+ public String[] listTables(String catalogName, String databaseName) {
+ return tableEnv.listTables(catalogName, databaseName);
}
@Override
- public void createTemporaryFunction(String path, UserDefinedFunction functionInstance) {
- delegate().createTemporaryFunction(path, functionInstance);
+ public String explainSql(
+ String statement, ExplainFormat format, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, format, extraDetails);
}
@Override
- public boolean dropTemporaryFunction(String path) {
- return delegate().dropTemporaryFunction(path);
+ public CompiledPlan loadPlan(PlanReference planReference) throws TableException {
+ return tableEnv.loadPlan(planReference);
}
@Override
- public void createTemporaryView(String path, Table view) {
- delegate().createTemporaryView(path, view);
+ public CompiledPlan compilePlanSql(String statement) throws TableException {
+ return tableEnv.compilePlanSql(statement);
}
@Override
- public Table from(String path) {
- return delegate().from(path);
+ public void registerCatalog(String catalogName, Catalog catalog) {
+ tableEnv.registerCatalog(catalogName, catalog);
}
@Override
- public String[] listCatalogs() {
- return delegate().listCatalogs();
+ public Optional<Catalog> getCatalog(String catalogName) {
+ return tableEnv.getCatalog(catalogName);
}
@Override
- public String[] listModules() {
- return delegate().listModules();
+ public void loadModule(String moduleName, Module module) {
+ tableEnv.loadModule(moduleName, module);
}
@Override
- public String[] listDatabases() {
- return delegate().listDatabases();
+ public void unloadModule(String moduleName) {
+ tableEnv.unloadModule(moduleName);
}
@Override
- public String[] listTables() {
- return delegate().listTables();
+ public void createTemporarySystemFunction(
+ String name, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createTemporarySystemFunction(name, functionClass);
}
@Override
- public String[] listViews() {
- return delegate().listViews();
+ public void createTemporarySystemFunction(
+ String name, UserDefinedFunction functionInstance) {
+ tableEnv.createTemporarySystemFunction(name, functionInstance);
}
@Override
- public String[] listTemporaryTables() {
- return delegate().listTemporaryTables();
+ public boolean dropTemporarySystemFunction(String name) {
+ return tableEnv.dropTemporarySystemFunction(name);
}
@Override
- public String[] listTemporaryViews() {
- return delegate().listTemporaryViews();
+ public void createFunction(String path, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createFunction(path, functionClass);
}
@Override
- public String[] listUserDefinedFunctions() {
- return delegate().listUserDefinedFunctions();
+ public void createFunction(
+ String path,
+ Class<? extends UserDefinedFunction> functionClass,
+ boolean ignoreIfExists) {
+ tableEnv.createFunction(path, functionClass, ignoreIfExists);
}
@Override
- public String[] listFunctions() {
- return delegate().listFunctions();
+ public boolean dropFunction(String path) {
+ return tableEnv.dropFunction(path);
}
@Override
- public boolean dropTemporaryTable(String path) {
- return delegate().dropTemporaryTable(path);
+ public void createTemporaryFunction(
+ String path, Class<? extends UserDefinedFunction> functionClass) {
+ tableEnv.createTemporaryFunction(path, functionClass);
}
@Override
- public boolean dropTemporaryView(String path) {
- return delegate().dropTemporaryView(path);
+ public void createTemporaryFunction(String path, UserDefinedFunction functionInstance) {
+ tableEnv.createTemporaryFunction(path, functionInstance);
}
@Override
- public String explainSql(String statement, ExplainDetail... extraDetails) {
- return delegate().explainSql(statement, extraDetails);
+ public boolean dropTemporaryFunction(String path) {
+ return tableEnv.dropTemporaryFunction(path);
}
@Override
- public Table sqlQuery(String query) {
- return delegate().sqlQuery(query);
+ public void createTemporaryView(String path, Table view) {
+ tableEnv.createTemporaryView(path, view);
}
@Override
- public TableResult executeSql(String statement) {
- return delegate().executeSql(statement);
+ public Table from(String path) {
+ return tableEnv.from(path);
}
@Override
- public String getCurrentCatalog() {
- return delegate().getCurrentCatalog();
+ public String[] listCatalogs() {
+ return tableEnv.listCatalogs();
}
@Override
- public void useCatalog(String catalogName) {
- delegate().useCatalog(catalogName);
+ public String[] listModules() {
+ return tableEnv.listModules();
}
@Override
- public String getCurrentDatabase() {
- return delegate().getCurrentDatabase();
+ public String[] listDatabases() {
+ return tableEnv.listDatabases();
}
@Override
- public void useDatabase(String databaseName) {
- delegate().useDatabase(databaseName);
+ public String[] listTables() {
+ return tableEnv.listTables();
}
@Override
- public TableConfig getConfig() {
- return delegate().getConfig();
+ public String[] listViews() {
+ return tableEnv.listViews();
}
@Override
- public StatementSet createStatementSet() {
- return delegate().createStatementSet();
+ public String[] listTemporaryTables() {
+ return tableEnv.listTemporaryTables();
}
@Override
- public void useModules(String... modules) {
- delegate().useModules(modules);
+ public String[] listTemporaryViews() {
+ return tableEnv.listTemporaryViews();
}
@Override
- public ModuleEntry[] listFullModules() {
- return delegate().listFullModules();
+ public String[] listUserDefinedFunctions() {
+ return tableEnv.listUserDefinedFunctions();
}
@Override
- public void createTemporaryTable(String path, TableDescriptor descriptor) {
- delegate().createTemporaryTable(path, descriptor);
+ public String[] listFunctions() {
+ return tableEnv.listFunctions();
}
@Override
- public void createTable(String path, TableDescriptor descriptor) {
- delegate().createTable(path, descriptor);
+ public boolean dropTemporaryTable(String path) {
+ return tableEnv.dropTemporaryTable(path);
}
@Override
- public Table from(TableDescriptor descriptor) {
- return delegate().from(descriptor);
+ public boolean dropTemporaryView(String path) {
+ return tableEnv.dropTemporaryView(path);
}
@Override
- public void registerFunction(String name, ScalarFunction function) {
- delegate().registerFunction(name, function);
+ public String explainSql(String statement, ExplainDetail... extraDetails) {
+ return tableEnv.explainSql(statement, extraDetails);
}
@Override
- public void registerTable(String name, Table table) {
- delegate().registerTable(name, table);
+ public Table sqlQuery(String query) {
+ return tableEnv.sqlQuery(query);
}
@Override
- public Table scan(String... tablePath) {
- return delegate().scan(tablePath);
+ public TableResult executeSql(String statement) {
+ return tableEnv.executeSql(statement);
}
@Override
- public String[] getCompletionHints(String statement, int position) {
- return delegate().getCompletionHints(statement, position);
+ public String getCurrentCatalog() {
+ return tableEnv.getCurrentCatalog();
}
@Override
- public String[] listTables(String catalogName, String databaseName) {
- return delegate().listTables(catalogName, databaseName);
+ public void useCatalog(String catalogName) {
+ tableEnv.useCatalog(catalogName);
}
@Override
- public CompiledPlan loadPlan(PlanReference planReference) {
- return delegate().loadPlan(planReference);
+ public String getCurrentDatabase() {
+ return tableEnv.getCurrentDatabase();
}
@Override
- public CompiledPlan compilePlanSql(String stmt) {
- return delegate().compilePlanSql(stmt);
+ public void useDatabase(String databaseName) {
+ tableEnv.useDatabase(databaseName);
}
@Override
- public void createFunction(String path, String className, List<ResourceUri> resourceUris) {
- delegate().createFunction(path, className, resourceUris);
+ public TableConfig getConfig() {
+ return tableEnv.getConfig();
}
@Override
- public void createFunction(
- String path, String className, List<ResourceUri> resourceUris, boolean ignoreIfExists) {
- delegate().createFunction(path, className, resourceUris, ignoreIfExists);
+ public StatementSet createStatementSet() {
+ return tableEnv.createStatementSet();
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createTemporaryFunction(String path, String className, List<ResourceUri> resourceUris) {
- delegate().createTemporaryFunction(path, className, resourceUris);
+ public void registerFunction(String name, ScalarFunction function) {
+ tableEnv.registerFunction(name, function);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createTemporarySystemFunction(String name, String className, List<ResourceUri> resourceUris) {
- delegate().createTemporarySystemFunction(name, className, resourceUris);
+ public void registerTable(String name, Table table) {
+ tableEnv.registerTable(name, table);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public String explainSql(String statement, ExplainFormat format, ExplainDetail... extraDetails) {
- return delegate().explainSql(statement, format, extraDetails);
+ public Table scan(String... tablePath) {
+ return tableEnv.scan(tablePath);
}
+ /** @deprecated Retained for backward compatibility with legacy Flink Table API. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
@Override
- public void createCatalog(String catalog, CatalogDescriptor catalogDescriptor) {
- delegate().createCatalog(catalog, catalogDescriptor);
+ public String[] getCompletionHints(String statement, int position) {
+ return tableEnv.getCompletionHints(statement, position);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
index fef182572..dd8ccad74 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommand.java
@@ -20,14 +20,16 @@ package org.apache.streampark.flink.core;
import org.apache.commons.lang3.StringUtils;
import java.util.Optional;
-import java.util.function.Function;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-/** Supported Flink SQL command types. */
+/** Flink SQL command types. */
public enum SqlCommand {
+ // ---- SELECT Statements -----------------------------------------------------------------
SELECT("select", "(SELECT\\s+.+)"),
+
+ // ---- CREATE Statements -----------------------------------------------------------------
CREATE_TABLE("create table", "(CREATE\\s+(TEMPORARY\\s+|)TABLE\\s+.+)"),
CREATE_CATALOG("create catalog", "(CREATE\\s+CATALOG\\s+.+)"),
CREATE_DATABASE("create database", "(CREATE\\s+DATABASE\\s+.+)"),
@@ -37,26 +39,39 @@ public enum SqlCommand {
CREATE_FUNCTION(
"create function",
"(CREATE\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+(IF\\s+NOT\\s+EXISTS\\s+|)(\\S+)\\s+AS\\s+.*)"),
+
+ // ---- DROP Statements -------------------------------------------------------------------
DROP_CATALOG("drop catalog", "(DROP\\s+CATALOG\\s+.+)"),
DROP_TABLE("drop table", "(DROP\\s+(TEMPORARY\\s+|)TABLE\\s+.+)"),
DROP_DATABASE("drop database", "(DROP\\s+DATABASE\\s+.+)"),
DROP_VIEW("drop view", "(DROP\\s+(TEMPORARY\\s+|)VIEW\\s+.+)"),
DROP_FUNCTION(
- "drop function",
- "(DROP\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+ "drop function", "(DROP\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+
+ // ---- ALTER Statements ------------------------------------------------------------------
ALTER_TABLE("alter table", "(ALTER\\s+TABLE\\s+.+)"),
ALTER_VIEW("alter view", "(ALTER\\s+VIEW\\s+.+)"),
ALTER_DATABASE("alter database", "(ALTER\\s+DATABASE\\s+.+)"),
ALTER_FUNCTION(
"alter function",
"(ALTER\\s+(TEMPORARY\\s+|TEMPORARY\\s+SYSTEM\\s+|)FUNCTION\\s+.+)"),
+
+ // ---- INSERT Statement ------------------------------------------------------------------
INSERT("insert", "(INSERT\\s+(INTO|OVERWRITE)\\s+.+)"),
+
+ // ---- DESCRIBE Statement ----------------------------------------------------------------
DESC("desc", "(DESC\\s+.+)"),
DESCRIBE("describe", "(DESCRIBE\\s+.+)"),
+
+ // ---- EXPLAIN Statement -----------------------------------------------------------------
EXPLAIN("explain", "(EXPLAIN\\s+.+)"),
+
+ // ---- USE Statements --------------------------------------------------------------------
USE_CATALOG("use catalog", "(USE\\s+CATALOG\\s+.+)"),
USE_MODULES("use modules", "(USE\\s+MODULES\\s+.+)"),
USE_DATABASE("use database", "(USE\\s+(?!(CATALOG|MODULES)).+)"),
+
+ // ---- SHOW Statements -------------------------------------------------------------------
SHOW_CATALOGS("show catalogs", "(SHOW\\s+CATALOGS\\s*)"),
SHOW_CURRENT_CATALOG("show current catalog", "(SHOW\\s+CURRENT\\s+CATALOG\\s*)"),
SHOW_DATABASES("show databases", "(SHOW\\s+DATABASES\\s*)"),
@@ -68,65 +83,88 @@ public enum SqlCommand {
SHOW_CREATE_VIEW("show create view", "(SHOW\\s+CREATE\\s+VIEW\\s+.+)"),
SHOW_FUNCTIONS("show functions", "(SHOW\\s+(USER\\s+|)FUNCTIONS\\s*)"),
SHOW_MODULES("show modules", "(SHOW\\s+(FULL\\s+|)MODULES\\s*)"),
+
+ // ---- LOAD Statements -------------------------------------------------------------------
LOAD_MODULE("load module", "(LOAD\\s+MODULE\\s+.+)"),
+
+ // ---- UNLOAD Statements -----------------------------------------------------------------
UNLOAD_MODULE("unload module", "(UNLOAD\\s+MODULE\\s+.+)"),
- SET("set", "SET(\\s+(\\S+)\\s*=(.*))?", SqlCommandConverters::setOperands),
+
+ // ---- SET Statements --------------------------------------------------------------------
+ SET(
+ "set",
+ "SET(\\s+(\\S+)\\s*=(.*))?",
+ groups -> {
+ if (groups.length < 3) {
+ return Optional.empty();
+ }
+ if (groups[0] == null) {
+ return Optional.of(new String[]{cleanUp(groups[0])});
+ }
+ return Optional.of(new String[]{cleanUp(groups[1]), cleanUp(groups[2])});
+ }),
+
+ // ---- RESET Statements ------------------------------------------------------------------
RESET("reset", "RESET\\s+'(.*)'"),
- RESET_ALL("reset all", "RESET", SqlCommandConverters::resetAll),
+ RESET_ALL("reset all", "RESET", groups -> Optional.of(new String[]{"ALL"})),
+
+ // ---- INSERT SET Statements -------------------------------------------------------------
+ /** @deprecated SQL Client syntax; not supported on this platform. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
BEGIN_STATEMENT_SET(
- "begin statement set", "BEGIN\\s+STATEMENT\\s+SET", SqlCommandConverters::noOperands),
- END_STATEMENT_SET("end statement set", "END", SqlCommandConverters::noOperands),
+ "begin statement set", "BEGIN\\s+STATEMENT\\s+SET", SqlCommandConverters.NO_OPERANDS),
+ /** @deprecated SQL Client syntax; not supported on this platform. */
+ @Deprecated(since = "2.1.0", forRemoval = false)
+ END_STATEMENT_SET("end statement set", "END", SqlCommandConverters.NO_OPERANDS),
+
+ // Since: 2.1.2 for flink 1.18
DELETE("delete", "(DELETE\\s+FROM\\s+.+)"),
UPDATE("update", "(UPDATE\\s+.+)");
- private final String commandName;
+ private static final int PATTERN_FLAGS = Pattern.CASE_INSENSITIVE | Pattern.DOTALL;
+
+ private final String name;
private final String regex;
- private final Function<String[], Optional<String[]>> converter;
+ private final SqlCommandConverter converter;
private Matcher matcher;
- SqlCommand(String commandName, String regex) {
- this(commandName, regex, SqlCommandConverters::firstGroup);
+ SqlCommand(String name, String regex) {
+ this(name, regex, SqlCommandConverters.DEFAULT);
}
- SqlCommand(
- String commandName,
- String regex,
- Function<String[], Optional<String[]>> converter) {
- this.commandName = commandName;
+ SqlCommand(String name, String regex, SqlCommandConverter converter) {
+ this.name = name;
this.regex = regex;
this.converter = converter;
}
- /** Scala field-style access (lowercase command label). */
- public String commandName() {
- return commandName;
+ /** Command label (e.g. {@code "select"}, {@code "create table"}). */
+ public String getName() {
+ return name;
}
- public String getCommandName() {
- return commandName;
+ public String getRegex() {
+ return regex;
}
- public Matcher matcher() {
- return matcher;
+ public SqlCommandConverter getConverter() {
+ return converter;
}
public Matcher getMatcher() {
return matcher;
}
- public Optional<String[]> convert(String[] groups) {
- return converter.apply(groups);
- }
-
public boolean matches(String input) {
if (StringUtils.isBlank(regex)) {
return false;
}
- Pattern pattern = Pattern.compile(regex, Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
+ Pattern pattern = Pattern.compile(regex, PATTERN_FLAGS);
matcher = pattern.matcher(input);
return matcher.matches();
}
+ /** Resolve the first matching command for the given statement. */
public static SqlCommand get(String stmt) {
for (SqlCommand command : values()) {
if (command.matches(stmt)) {
@@ -135,4 +173,16 @@ public enum SqlCommand {
}
return null;
}
+
+ static String cleanUp(String sql) {
+ String trimmed = sql.trim();
+ if (trimmed.length() >= 2) {
+ char first = trimmed.charAt(0);
+ char last = trimmed.charAt(trimmed.length() - 1);
+ if ((first == '\'' && last == '\'') || (first == '"' && last == '"')) {
+ return trimmed.substring(1, trimmed.length() - 1);
+ }
+ }
+ return trimmed;
+ }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandCall.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandCall.java
index a39083c01..cb16ed5b5 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandCall.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandCall.java
@@ -20,11 +20,11 @@ package org.apache.streampark.flink.core;
/** Call of SQL command with operands and command type. */
public class SqlCommandCall {
- private final int lineStart;
- private final int lineEnd;
- private final SqlCommand command;
- private final String[] operands;
- private final String originSql;
+ public final int lineStart;
+ public final int lineEnd;
+ public final SqlCommand command;
+ public final String[] operands;
+ public final String originSql;
public SqlCommandCall(
int lineStart,
@@ -38,44 +38,4 @@ public class SqlCommandCall {
this.operands = operands;
this.originSql = originSql;
}
-
- public int lineStart() {
- return lineStart;
- }
-
- public int getLineStart() {
- return lineStart;
- }
-
- public int lineEnd() {
- return lineEnd;
- }
-
- public int getLineEnd() {
- return lineEnd;
- }
-
- public SqlCommand command() {
- return command;
- }
-
- public SqlCommand getCommand() {
- return command;
- }
-
- public String[] operands() {
- return operands;
- }
-
- public String[] getOperands() {
- return operands;
- }
-
- public String originSql() {
- return originSql;
- }
-
- public String getOriginSql() {
- return originSql;
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.14/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverter.java
similarity index 79%
rename from streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.14/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
rename to streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverter.java
index 4914c46e9..91e885305 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.14/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverter.java
@@ -17,11 +17,11 @@
package org.apache.streampark.flink.core;
-import org.apache.flink.client.program.ClusterClient;
+import java.util.Optional;
-public class FlinkClusterClient<T> extends FlinkClientTrait<T> {
+/** Converts regex capture groups into command operands. */
+@FunctionalInterface
+public interface SqlCommandConverter {
- public FlinkClusterClient(ClusterClient<T> clusterClient) {
- super(clusterClient);
- }
+ Optional<String[]> convert(String[] groups);
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverters.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverters.java
index b787ba9b6..1ff870456 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverters.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandConverters.java
@@ -19,50 +19,17 @@ package org.apache.streampark.flink.core;
import java.util.Optional;
-/** Operand converters for {@link SqlCommand}. */
-final class SqlCommandConverters {
+/** SQL command operand converters. */
+public final class SqlCommandConverters {
private SqlCommandConverters() {
}
- static Optional<String[]> noOperands(String[] groups) {
- if (groups == null) {
- return Optional.empty();
- }
- return Optional.of(new String[0]);
- }
-
- static Optional<String[]> firstGroup(String[] groups) {
- return Optional.of(new String[]{groups[0]});
- }
-
- static Optional<String[]> setOperands(String[] groups) {
- if (groups.length < 3) {
- return Optional.empty();
- }
- if (groups[0] == null) {
- return Optional.of(new String[]{cleanUp(groups[0])});
- }
- return Optional.of(new String[]{cleanUp(groups[1]), cleanUp(groups[2])});
- }
-
- static Optional<String[]> resetAll(String[] groups) {
- if (groups == null) {
- return Optional.empty();
- }
- return Optional.of(new String[]{"ALL"});
- }
+ /** Converter that produces no operands. */
+ public static final SqlCommandConverter NO_OPERANDS =
+ groups -> Optional.of(new String[0]);
- private static String cleanUp(String sql) {
- if (sql == null) {
- return null;
- }
- String trimmed = sql.trim();
- if (trimmed.length() >= 2
- && ((trimmed.startsWith("'") && trimmed.endsWith("'"))
- || (trimmed.startsWith("\"") && trimmed.endsWith("\"")))) {
- return trimmed.substring(1, trimmed.length() - 1);
- }
- return trimmed;
- }
+ /** Default converter that uses the first capture group as the sole operand. */
+ public static final SqlCommandConverter DEFAULT =
+ groups -> Optional.of(new String[]{groups[0]});
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandParser.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandParser.java
index a58bbf8e6..15e991a2f 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandParser.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlCommandParser.java
@@ -26,64 +26,72 @@ import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.function.Consumer;
+import java.util.regex.Matcher;
-/** Parses Flink SQL scripts into {@link SqlCommandCall} instances. */
-public final class SqlCommandParser extends org.apache.streampark.common.util.LoggerSupport {
+/** Parser for Flink SQL commands. */
+public final class SqlCommandParser {
- private SqlCommandParser() {
- }
+ private static final String SQL_EMPTY_ERROR = "verify failed: flink sql cannot be empty.";
- public static List<SqlCommandCall> parseSQL(String sql) {
- return parseSQL(sql, null);
+ private SqlCommandParser() {
}
- public static List<SqlCommandCall> parseSQL(
- String sql,
- Consumer<FlinkSqlValidationResult> validationCallback) {
- String sqlEmptyError = "verify failed: flink sql cannot be empty.";
+ public static List<SqlCommandCall> parseSQL(String sql, Consumer<FlinkSqlValidationResult> callback) {
if (StringUtils.isBlank(sql)) {
- throw new IllegalArgumentException(sqlEmptyError);
+ if (callback != null) {
+ callback.accept(
+ FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
+ .exception(SQL_EMPTY_ERROR)
+ .build());
+ return Collections.emptyList();
+ }
+ throw new IllegalArgumentException(SQL_EMPTY_ERROR);
}
+
List<SqlSegment> sqlSegments = SqlSplitter.splitSql(sql);
if (sqlSegments.isEmpty()) {
- if (validationCallback != null) {
- validationCallback.accept(
- new FlinkSqlValidationResult()
- .withSuccess(false)
- .withFailedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
- .withException(sqlEmptyError));
+ if (callback != null) {
+ callback.accept(
+ FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
+ .exception(SQL_EMPTY_ERROR)
+ .build());
return Collections.emptyList();
}
- throw new IllegalArgumentException(sqlEmptyError);
+ throw new IllegalArgumentException(SQL_EMPTY_ERROR);
}
List<SqlCommandCall> calls = new ArrayList<>();
for (SqlSegment segment : sqlSegments) {
- Optional<SqlCommandCall> call = parseLine(segment);
- if (call.isPresent()) {
- calls.add(call.get());
- } else if (validationCallback != null) {
- validationCallback.accept(
- new FlinkSqlValidationResult()
- .withSuccess(false)
- .withFailedType(FlinkSqlValidationFailedType.UNSUPPORTED_SQL)
- .withLineStart(segment.start())
- .withLineEnd(segment.end())
- .withException("unsupported sql")
- .withSql(segment.sql()));
- return Collections.emptyList();
+ Optional<SqlCommandCall> parsed = parseLine(segment);
+ if (parsed.isPresent()) {
+ calls.add(parsed.get());
+ } else if (callback != null) {
+ callback.accept(
+ FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.UNSUPPORTED_SQL)
+ .lineStart(segment.start)
+ .lineEnd(segment.end)
+ .exception("unsupported sql")
+ .sql(segment.sql)
+ .build());
} else {
- throw new UnsupportedOperationException("unsupported sql: " + segment.sql());
+ throw new UnsupportedOperationException("unsupported sql: " + segment.sql);
}
}
if (calls.isEmpty()) {
- if (validationCallback != null) {
- validationCallback.accept(
- new FlinkSqlValidationResult()
- .withSuccess(false)
- .withFailedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
- .withException("flink sql syntax error, no executable sql"));
+ if (callback != null) {
+ callback.accept(
+ FlinkSqlValidationResult.builder()
+ .success(false)
+ .failedType(FlinkSqlValidationFailedType.VERIFY_FAILED)
+ .exception("flink sql syntax error, no executable sql")
+ .build());
return Collections.emptyList();
}
throw new UnsupportedOperationException("flink sql syntax error, no executable sql");
@@ -92,22 +100,26 @@ public final class SqlCommandParser extends org.apache.streampark.common.util.Lo
}
private static Optional<SqlCommandCall> parseLine(SqlSegment sqlSegment) {
- SqlCommand sqlCommand = SqlCommand.get(sqlSegment.sql().trim());
+ SqlCommand sqlCommand = SqlCommand.get(sqlSegment.sql.trim());
if (sqlCommand == null) {
return Optional.empty();
}
- String[] groups = new String[sqlCommand.matcher().groupCount()];
+
+ Matcher matcher = sqlCommand.getMatcher();
+ String[] groups = new String[matcher.groupCount()];
for (int i = 0; i < groups.length; i++) {
- groups[i] = sqlCommand.matcher().group(i + 1);
+ groups[i] = matcher.group(i + 1);
}
+
return sqlCommand
+ .getConverter()
.convert(groups)
.map(
operands -> new SqlCommandCall(
- sqlSegment.start(),
- sqlSegment.end(),
+ sqlSegment.start,
+ sqlSegment.end,
sqlCommand,
operands,
- sqlSegment.sql().trim()));
+ sqlSegment.sql.trim()));
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSegment.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSegment.java
index a04f52f54..d2868cc14 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSegment.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSegment.java
@@ -17,40 +17,16 @@
package org.apache.streampark.flink.core;
-/** A segment of SQL text with line range metadata. */
+/** SQL segment with line range and content. */
public class SqlSegment {
- private final int start;
- private final int end;
- private final String sql;
+ public final int start;
+ public final int end;
+ public final String sql;
public SqlSegment(int start, int end, String sql) {
this.start = start;
this.end = end;
this.sql = sql;
}
-
- public int start() {
- return start;
- }
-
- public int getStart() {
- return start;
- }
-
- public int end() {
- return end;
- }
-
- public int getEnd() {
- return end;
- }
-
- public String sql() {
- return sql;
- }
-
- public String getSql() {
- return sql;
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSplitter.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSplitter.java
index af2c8c126..1f148e74d 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSplitter.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/SqlSplitter.java
@@ -31,8 +31,8 @@ import java.util.Map;
import java.util.Scanner;
import java.util.Set;
-/** Splits multi-statement Flink SQL scripts into segments with line numbers. */
-final class SqlSplitter {
+/** Splits multi-statement Flink SQL scripts into individual segments. */
+public final class SqlSplitter {
private static final Set<String> SINGLE_LINE_COMMENT_PREFIX_LIST;
@@ -45,10 +45,117 @@ final class SqlSplitter {
private SqlSplitter() {
}
- static List<SqlSegment> splitSql(String sql) {
- QueryExtractor extractor = new QueryExtractor(sql);
- Map<Integer, String> refinedQueries = refineQueries(extractor.queries());
- return buildSegments(refinedQueries, extractor.lineNumMap());
+ /**
+ * Split whole text into multiple sql statements. Two Steps: Step 1, split the whole text into
+ * multiple sql statements. Step 2, refine the results. Replace the preceding sql statements with
+ * empty lines, so that we can get the correct line number in the parsing error message.
+ */
+ public static List<SqlSegment> splitSql(String sql) {
+ Map<Integer, Boolean> lineDescriptor = buildLineDescriptor(sql);
+ Map<Integer, int[]> lineNumMap = new HashMap<>();
+ List<String> queries = splitRawQueries(sql, lineDescriptor, lineNumMap);
+ Map<Integer, String> refinedQueries = refineQueries(queries);
+ return toSegments(refinedQueries, lineNumMap);
+ }
+
+ private static List<String> splitRawQueries(
+ String sql, Map<Integer, Boolean> lineDescriptor,
+ Map<Integer, int[]> lineNumMap) {
+ List<String> queries = new ArrayList<>();
+ if (StringUtils.isBlank(sql)) {
+ return queries;
+ }
+
+ int lastIndex = sql.length() - 1;
+ StringBuilder query = new StringBuilder();
+ ParseState state = new ParseState();
+
+ for (int idx = 0; idx < sql.length(); idx++) {
+ if (sql.charAt(idx) == '\n') {
+ state.lineNum++;
+ }
+ char ch = sql.charAt(idx);
+ processCharacter(
+ sql, idx, lastIndex, ch, query, queries, state, lineDescriptor, lineNumMap);
+ }
+ return queries;
+ }
+
+ private static void processCharacter(
+ String sql,
+ int idx,
+ int lastIndex,
+ char ch,
+ StringBuilder query,
+ List<String> queries,
+ ParseState state,
+ Map<Integer, Boolean> lineDescriptor,
+ Map<Integer, int[]> lineNumMap) {
+ if (state.endSingleLineComment(ch)) {
+ query.append(ch);
+ appendTrailingQuery(query, queries, idx, lastIndex);
+ return;
+ }
+
+ state.endMultiLineComment(sql, idx);
+ state.toggleQuoteState(ch, idx);
+ state.startCommentIfNeeded(sql, idx, lastIndex);
+
+ if (state.isStatementDelimiter(ch)) {
+ finishStatement(query, queries, state, lineDescriptor, lineNumMap);
+ return;
+ }
+
+ if (idx == lastIndex) {
+ finishLastCharacter(sql, idx, lastIndex, ch, query, queries, state, lineDescriptor, lineNumMap);
+ return;
+ }
+
+ if (state.shouldAppendChar(ch)) {
+ query.append(ch);
+ } else if (ch == '\n') {
+ query.append(ch);
+ }
+ }
+
+ private static void appendTrailingQuery(
+ StringBuilder query, List<String> queries, int idx, int lastIndex) {
+ if (idx == lastIndex && StringUtils.isNotBlank(query.toString().trim())) {
+ queries.add(query.toString());
+ }
+ }
+
+ private static void finishStatement(
+ StringBuilder query,
+ List<String> queries,
+ ParseState state,
+ Map<Integer, Boolean> lineDescriptor,
+ Map<Integer, int[]> lineNumMap) {
+ markLineNumber(state.lineNum, lineDescriptor, lineNumMap);
+ if (StringUtils.isNotBlank(query.toString().trim())) {
+ queries.add(query.toString());
+ }
+ query.setLength(0);
+ }
+
+ private static void finishLastCharacter(
+ String sql,
+ int idx,
+ int lastIndex,
+ char ch,
+ StringBuilder query,
+ List<String> queries,
+ ParseState state,
+ Map<Integer, Boolean> lineDescriptor,
+ Map<Integer, int[]> lineNumMap) {
+ markLineNumber(state.lineNum, lineDescriptor, lineNumMap);
+ if (!state.singleLineComment && !state.multiLineComment) {
+ query.append(ch);
+ }
+ if (StringUtils.isNotBlank(query.toString().trim())) {
+ queries.add(query.toString());
+ }
+ query.setLength(0);
}
private static Map<Integer, String> refineQueries(List<String> queries) {
@@ -56,201 +163,77 @@ final class SqlSplitter {
for (int i = 0; i < queries.size(); i++) {
String currStatement = queries.get(i);
if (isSingleLineComment(currStatement) || isMultipleLineComment(currStatement)) {
- mergeCommentIntoPrevious(refinedQueries, currStatement);
+ appendCommentLineBreaks(refinedQueries, currStatement);
} else {
- appendStatement(refinedQueries, i, currStatement);
+ refinedQueries.put(
+ refinedQueries.size(),
+ leadingLineBreaks(refinedQueries, i) + currStatement);
}
}
return refinedQueries;
}
- private static void mergeCommentIntoPrevious(Map<Integer, String> refinedQueries, String comment) {
- if (!refinedQueries.isEmpty()) {
- int lastKey = refinedQueries.size() - 1;
- refinedQueries.put(lastKey, refinedQueries.get(lastKey) + extractLineBreaks(comment));
- }
- }
-
- private static void appendStatement(
- Map<Integer, String> refinedQueries,
- int index,
- String statement) {
- String linesPlaceholder = "";
- if (index > 0) {
- linesPlaceholder = extractLineBreaks(refinedQueries.get(index - 1));
+ private static void appendCommentLineBreaks(Map<Integer, String> refinedQueries, String currStatement) {
+ if (refinedQueries.isEmpty()) {
+ return;
}
- refinedQueries.put(refinedQueries.size(), linesPlaceholder + statement);
+ int lastKey = refinedQueries.size() - 1;
+ refinedQueries.put(lastKey, refinedQueries.get(lastKey) + extractLineBreaks(currStatement));
}
- private static String extractLineBreaks(String text) {
- StringBuilder builder = new StringBuilder();
- for (int i = 0; i < text.length(); i++) {
- if (text.charAt(i) == '\n') {
- builder.append('\n');
- }
+ private static String leadingLineBreaks(Map<Integer, String> refinedQueries, int index) {
+ if (index == 0) {
+ return "";
}
- return builder.toString();
+ return extractLineBreaks(refinedQueries.get(index - 1));
}
- private static List<SqlSegment> buildSegments(
- Map<Integer, String> refinedQueries,
- Map<Integer, int[]> lineNumMap) {
+ private static List<SqlSegment> toSegments(
+ Map<Integer, String> refinedQueries, Map<Integer, int[]> lineNumMap) {
List<SqlSegment> segments = new ArrayList<>();
for (Map.Entry<Integer, String> entry : refinedQueries.entrySet()) {
int[] line = lineNumMap.get(entry.getKey());
segments.add(new SqlSegment(line[0], line[1], entry.getValue()));
}
- segments.sort(Comparator.comparingInt(SqlSegment::start));
+ segments.sort(Comparator.comparingInt(a -> a.start));
return segments;
}
- private static boolean isSingleLineComment(String text) {
- return text.trim().startsWith(ConfigKeys.PARAM_PREFIX());
- }
-
- private static boolean isMultipleLineComment(String text) {
- return text.trim().startsWith("/*") && text.trim().endsWith("*/");
- }
-
- private static boolean isSingleLineComment(char curChar, char nextChar) {
- for (String prefix : SINGLE_LINE_COMMENT_PREFIX_LIST) {
- if (prefix.length() == 1 && curChar == prefix.charAt(0)) {
- return true;
- }
- if (prefix.length() == 2
- && curChar == prefix.charAt(0)
- && nextChar == prefix.charAt(1)) {
- return true;
- }
- }
- return false;
- }
-
- private static final class QueryExtractor {
+ private static final class ParseState {
- private final String sql;
- private final int lastIndex;
- private final Map<Integer, int[]> lineNumMap = new HashMap<>();
- private final Map<Integer, Boolean> lineDescriptor;
- private final List<String> queries = new ArrayList<>();
-
- private final StringBuilder query = new StringBuilder();
private boolean multiLineComment;
private boolean singleLineComment;
private boolean singleQuoteString;
private boolean doubleQuoteString;
private int lineNum;
- private QueryExtractor(String sql) {
- this.sql = sql;
- this.lastIndex = StringUtils.isNotBlank(sql) ? sql.length() - 1 : 0;
- this.lineDescriptor = buildLineDescriptor(sql);
- for (int idx = 0; idx < sql.length(); idx++) {
- processCharacter(idx);
- }
- }
-
- private List<String> queries() {
- return queries;
- }
-
- private Map<Integer, int[]> lineNumMap() {
- return lineNumMap;
- }
-
- private static Map<Integer, Boolean> buildLineDescriptor(String sql) {
- Map<Integer, Boolean> descriptor = new HashMap<>();
- Scanner scanner = new Scanner(sql);
- int lineNumber = 0;
- boolean startComment = false;
- boolean hasComment = false;
-
- while (scanner.hasNextLine()) {
- lineNumber++;
- String line = scanner.nextLine().trim();
- boolean nonEmpty =
- StringUtils.isNotBlank(line) && !line.startsWith(ConfigKeys.PARAM_PREFIX());
- if (line.startsWith("/*")) {
- startComment = true;
- hasComment = true;
- }
- descriptor.put(lineNumber, nonEmpty && !hasComment);
- if (startComment && line.endsWith("*/")) {
- startComment = false;
- hasComment = false;
- }
- }
- scanner.close();
- return descriptor;
- }
-
- private static int findStartLine(int num, Map<Integer, Boolean> lineDescriptor) {
- if (num >= lineDescriptor.size() || Boolean.TRUE.equals(lineDescriptor.get(num))) {
- return num;
- }
- return findStartLine(num + 1, lineDescriptor);
- }
-
- private static void markLineNumber(
- int lineNum,
- Map<Integer, int[]> lineNumMap,
- Map<Integer, Boolean> lineDescriptor) {
- int line = lineNum + 1;
- if (lineNumMap.isEmpty()) {
- lineNumMap.put(0, new int[]{findStartLine(1, lineDescriptor), line});
- } else {
- int index = lineNumMap.size();
- int start = lineNumMap.get(lineNumMap.size() - 1)[1] + 1;
- lineNumMap.put(index, new int[]{findStartLine(start, lineDescriptor), line});
- }
- }
-
- private static boolean hasNonBlankQuery(StringBuilder query) {
- return !query.toString().trim().isEmpty();
- }
-
- private void processCharacter(int idx) {
- if (sql.charAt(idx) == '\n') {
- lineNum++;
- }
- char ch = sql.charAt(idx);
- if (handleSingleLineCommentEnd(idx, ch)) {
- return;
- }
- updateMultiLineCommentEnd(idx);
- updateQuoteState(idx, ch);
- updateCommentStart(idx);
- handleStatementBoundary(idx, ch);
- }
-
- private boolean handleSingleLineCommentEnd(int idx, char ch) {
- if (!singleLineComment || ch != '\n') {
- return false;
- }
- singleLineComment = false;
- query.append(ch);
- if (idx == lastIndex && hasNonBlankQuery(query)) {
- queries.add(query.toString());
+ private boolean endSingleLineComment(char ch) {
+ if (singleLineComment && ch == '\n') {
+ singleLineComment = false;
+ return true;
}
- return true;
+ return false;
}
- private void updateMultiLineCommentEnd(int idx) {
- if (multiLineComment && idx - 1 >= 0 && sql.charAt(idx - 1) == '/'
- && idx - 2 >= 0 && sql.charAt(idx - 2) == '*') {
+ private void endMultiLineComment(String sql, int idx) {
+ if (multiLineComment
+ && idx - 1 >= 0
+ && sql.charAt(idx - 1) == '/'
+ && idx - 2 >= 0
+ && sql.charAt(idx - 2) == '*') {
multiLineComment = false;
}
}
- private void updateQuoteState(int idx, char ch) {
- if (ch == '\'' && !singleLineComment && !multiLineComment) {
+ private void toggleQuoteState(char ch, int idx) {
+ if (ch == '\'' && !(singleLineComment || multiLineComment)) {
if (singleQuoteString) {
singleQuoteString = false;
} else if (!doubleQuoteString) {
singleQuoteString = true;
}
}
- if (ch == '"' && !singleLineComment && !multiLineComment) {
+ if (ch == '"' && !(singleLineComment || multiLineComment)) {
if (doubleQuoteString && idx > 0) {
doubleQuoteString = false;
} else if (!singleQuoteString) {
@@ -259,8 +242,11 @@ final class SqlSplitter {
}
}
- private void updateCommentStart(int idx) {
- if (singleQuoteString || doubleQuoteString || multiLineComment || singleLineComment
+ private void startCommentIfNeeded(String sql, int idx, int lastIndex) {
+ if (singleQuoteString
+ || doubleQuoteString
+ || multiLineComment
+ || singleLineComment
|| idx >= lastIndex) {
return;
}
@@ -274,37 +260,104 @@ final class SqlSplitter {
}
}
- private void handleStatementBoundary(int idx, char ch) {
- if (ch == ';' && !singleQuoteString && !doubleQuoteString && !multiLineComment
- && !singleLineComment) {
- markLineNumber(lineNum, lineNumMap, lineDescriptor);
- if (hasNonBlankQuery(query)) {
- queries.add(query.toString());
- query.setLength(0);
- }
- return;
+ private boolean isStatementDelimiter(char ch) {
+ return ch == ';'
+ && !singleQuoteString
+ && !doubleQuoteString
+ && !multiLineComment
+ && !singleLineComment;
+ }
+
+ private boolean shouldAppendChar(char ch) {
+ return !singleLineComment && !multiLineComment;
+ }
+ }
+
+ private static Map<Integer, Boolean> buildLineDescriptor(String sql) {
+ Map<Integer, Boolean> descriptor = new HashMap<>();
+ Scanner scanner = new Scanner(sql);
+ int lineNumber = 0;
+ boolean startComment = false;
+ boolean hasComment = false;
+
+ while (scanner.hasNextLine()) {
+ lineNumber++;
+ String line = scanner.nextLine().trim();
+ boolean nonEmpty =
+ StringUtils.isNotBlank(line) && !line.startsWith(ConfigKeys.PARAM_PREFIX());
+ if (line.startsWith("/*")) {
+ startComment = true;
+ hasComment = true;
}
- if (idx == lastIndex) {
- handleLastCharacter(ch);
- return;
+
+ descriptor.put(lineNumber, nonEmpty && !hasComment);
+
+ if (startComment && line.endsWith("*/")) {
+ startComment = false;
+ hasComment = false;
}
- appendNonCommentCharacter(ch);
}
+ scanner.close();
+ return descriptor;
+ }
+
+ private static int findStartLine(int num, Map<Integer, Boolean> lineDescriptor) {
+ if (num >= lineDescriptor.size() || Boolean.TRUE.equals(lineDescriptor.get(num))) {
+ return num;
+ }
+ return findStartLine(num + 1, lineDescriptor);
+ }
- private void handleLastCharacter(char ch) {
- markLineNumber(lineNum, lineNumMap, lineDescriptor);
- if (!singleLineComment && !multiLineComment) {
- query.append(ch);
- }
- if (hasNonBlankQuery(query)) {
- queries.add(query.toString());
+ private static void markLineNumber(
+ int lineNum, Map<Integer, Boolean> lineDescriptor,
+ Map<Integer, int[]> lineNumMap) {
+ int line = lineNum + 1;
+ if (lineNumMap.isEmpty()) {
+ lineNumMap.put(0, new int[]{findStartLine(1, lineDescriptor), line});
+ } else {
+ int index = lineNumMap.size();
+ int[] previous = lineNumMap.get(lineNumMap.size() - 1);
+ int start = previous[1] + 1;
+ lineNumMap.put(index, new int[]{findStartLine(start, lineDescriptor), line});
+ }
+ }
+
+ private static String extractLineBreaks(String text) {
+ StringBuilder builder = new StringBuilder();
+ for (int i = 0; i < text.length(); i++) {
+ if (text.charAt(i) == '\n') {
+ builder.append('\n');
}
}
+ return builder.toString();
+ }
+
+ private static boolean isSingleLineComment(String text) {
+ return text.trim().startsWith(ConfigKeys.PARAM_PREFIX());
+ }
+
+ private static boolean isMultipleLineComment(String text) {
+ return text.trim().startsWith("/*") && text.trim().endsWith("*/");
+ }
- private void appendNonCommentCharacter(char ch) {
- if (!singleLineComment && !multiLineComment || ch == '\n') {
- query.append(ch);
+ private static boolean isSingleLineComment(char curChar, char nextChar) {
+ for (String singleCommentPrefix : SINGLE_LINE_COMMENT_PREFIX_LIST) {
+ switch (singleCommentPrefix.length()) {
+ case 1:
+ if (curChar == singleCommentPrefix.charAt(0)) {
+ return true;
+ }
+ break;
+ case 2:
+ if (curChar == singleCommentPrefix.charAt(0)
+ && nextChar == singleCommentPrefix.charAt(1)) {
+ return true;
+ }
+ break;
+ default:
+ break;
}
}
+ return false;
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamEnvConfig.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamEnvConfig.java
index 32bd6d745..4812573ff 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamEnvConfig.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamEnvConfig.java
@@ -17,31 +17,14 @@
package org.apache.streampark.flink.core;
+/** Stream execution environment configuration. */
public class StreamEnvConfig {
- private final String[] args;
- private final StreamEnvConfigFunction conf;
+ public final String[] args;
+ public final StreamEnvConfigFunction conf;
public StreamEnvConfig(String[] args, StreamEnvConfigFunction conf) {
this.args = args;
this.conf = conf;
}
-
- public String[] getArgs() {
- return args;
- }
-
- public StreamEnvConfigFunction getConf() {
- return conf;
- }
-
- /** Scala API alias for {@link #getArgs()}. */
- public String[] args() {
- return args;
- }
-
- /** Scala API alias for {@link #getConf()}. */
- public StreamEnvConfigFunction conf() {
- return conf;
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamTableEnvConfig.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamTableEnvConfig.java
index 8527dc70b..670fc3df2 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamTableEnvConfig.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/StreamTableEnvConfig.java
@@ -17,44 +17,19 @@
package org.apache.streampark.flink.core;
+/** Stream and table environment configuration. */
public class StreamTableEnvConfig {
- private final String[] args;
- private final StreamEnvConfigFunction streamConfig;
- private final TableEnvConfigFunction tableConfig;
+ public final String[] args;
+ public final StreamEnvConfigFunction streamConfig;
+ public final TableEnvConfigFunction tableConfig;
public StreamTableEnvConfig(
- String[] args, StreamEnvConfigFunction streamConfig,
+ String[] args,
+ StreamEnvConfigFunction streamConfig,
TableEnvConfigFunction tableConfig) {
this.args = args;
this.streamConfig = streamConfig;
this.tableConfig = tableConfig;
}
-
- public String[] getArgs() {
- return args;
- }
-
- public StreamEnvConfigFunction getStreamConfig() {
- return streamConfig;
- }
-
- public TableEnvConfigFunction getTableConfig() {
- return tableConfig;
- }
-
- /** Scala API alias for {@link #getArgs()}. */
- public String[] args() {
- return args;
- }
-
- /** Scala API alias for {@link #getStreamConfig()}. */
- public StreamEnvConfigFunction streamConfig() {
- return streamConfig;
- }
-
- /** Scala API alias for {@link #getTableConfig()}. */
- public TableEnvConfigFunction tableConfig() {
- return tableConfig;
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/TableEnvConfig.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/TableEnvConfig.java
index 18e0ef8e4..a9d6636d2 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/TableEnvConfig.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/TableEnvConfig.java
@@ -17,31 +17,14 @@
package org.apache.streampark.flink.core;
+/** Table environment configuration. */
public class TableEnvConfig {
- private final String[] args;
- private final TableEnvConfigFunction conf;
+ public final String[] args;
+ public final TableEnvConfigFunction conf;
public TableEnvConfig(String[] args, TableEnvConfigFunction conf) {
this.args = args;
this.conf = conf;
}
-
- public String[] getArgs() {
- return args;
- }
-
- public TableEnvConfigFunction getConf() {
- return conf;
- }
-
- /** Scala API alias for {@link #getArgs()}. */
- public String[] args() {
- return args;
- }
-
- /** Scala API alias for {@link #getConf()}. */
- public TableEnvConfigFunction conf() {
- return conf;
- }
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
index f269d77df..5bcfc40de 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkConfiguration.java
@@ -20,11 +20,12 @@ package org.apache.streampark.flink.core.conf;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.configuration.Configuration;
-public final class FlinkConfiguration {
+/** Flink runtime configuration holder. */
+public class FlinkConfiguration {
- private final ParameterTool parameter;
- private final Configuration envConfig;
- private final Configuration tableConfig;
+ public final ParameterTool parameter;
+ public final Configuration envConfig;
+ public final Configuration tableConfig;
public FlinkConfiguration(
ParameterTool parameter, Configuration envConfig, Configuration tableConfig) {
@@ -33,34 +34,7 @@ public final class FlinkConfiguration {
this.tableConfig = tableConfig;
}
- public ParameterTool getParameter() {
- return parameter;
- }
-
- public Configuration getEnvConfig() {
- return envConfig;
- }
-
- public Configuration getTableConfig() {
- return tableConfig;
- }
-
- /** Scala API alias for {@link #getParameter()}. */
- public ParameterTool parameter() {
- return parameter;
- }
-
- /** Scala API alias for {@link #getEnvConfig()}. */
- public Configuration envConfig() {
- return envConfig;
- }
-
- /** Scala API alias for {@link #getTableConfig()}. */
- public Configuration tableConfig() {
- return tableConfig;
- }
-
- public FlinkConfiguration withParameter(ParameterTool newParameter) {
- return new FlinkConfiguration(newParameter, envConfig, tableConfig);
+ public FlinkConfiguration withParameter(ParameterTool parameter) {
+ return new FlinkConfiguration(parameter, envConfig, tableConfig);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkRunOption.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkRunOption.java
index 5f838536f..4d0b66d0f 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkRunOption.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/FlinkRunOption.java
@@ -21,12 +21,11 @@ import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.DefaultParser;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
-/** Applies to all optional parameters under flink run */
+/** Applies to all optional parameters under flink run. */
public final class FlinkRunOption {
- private static final String SAVEPOINT_PATH = "savepointPath";
+ private static final String SAVEPOINT_PATH_ARG = "savepointPath";
public static final Option HELP_OPTION = new Option("h", "help", false, null);
public static final Option JAR_OPTION = new Option("j", "jarfile", true, null);
@@ -51,13 +50,12 @@ public final class FlinkRunOption {
public static final Option CANCEL_WITH_SAVEPOINT_OPTION =
new Option("s", "withSavepoint", true, null);
public static final Option STOP_WITH_SAVEPOINT_PATH =
- new Option("p", SAVEPOINT_PATH, true, null);
+ new Option("p", SAVEPOINT_PATH_ARG, true, null);
public static final Option STOP_AND_DRAIN = new Option("d", "drain", false, null);
public static final Option PY_OPTION = new Option("py", "python", true, null);
public static final Option PYFILES_OPTION = new Option("pyfs", "pyFiles", true, null);
public static final Option PYMODULE_OPTION = new Option("pym", "pyModule", true, null);
- public static final Option PYREQUIREMENTS_OPTION =
- new Option("pyreq", "pyRequirements", true, null);
+ public static final Option PYREQUIREMENTS_OPTION = new Option("pyreq", "pyRequirements", true, null);
public static final Option PYARCHIVE_OPTION = new Option("pyarch", "pyArchives", true, null);
public static final Option PYEXEC_OPTION = new Option("pyexec", "pyExecutable", true, null);
public static final Option EXECUTOR_OPTION = new Option("e", "executor", true, null);
@@ -97,7 +95,7 @@ public final class FlinkRunOption {
SCHEDULED_OPTION.setRequired(false);
SAVEPOINT_PATH_OPTION.setRequired(false);
- SAVEPOINT_PATH_OPTION.setArgName(SAVEPOINT_PATH);
+ SAVEPOINT_PATH_OPTION.setArgName(SAVEPOINT_PATH_ARG);
SAVEPOINT_ALLOW_NON_RESTORED_OPTION.setRequired(false);
@@ -109,7 +107,7 @@ public final class FlinkRunOption {
CANCEL_WITH_SAVEPOINT_OPTION.setOptionalArg(true);
STOP_WITH_SAVEPOINT_PATH.setRequired(false);
- STOP_WITH_SAVEPOINT_PATH.setArgName(SAVEPOINT_PATH);
+ STOP_WITH_SAVEPOINT_PATH.setArgName(SAVEPOINT_PATH_ARG);
STOP_WITH_SAVEPOINT_PATH.setOptionalArg(true);
STOP_AND_DRAIN.setRequired(false);
@@ -124,7 +122,9 @@ public final class FlinkRunOption {
PYMODULE_OPTION.setArgName("pythonModule");
PYREQUIREMENTS_OPTION.setRequired(false);
+
PYARCHIVE_OPTION.setRequired(false);
+
PYEXEC_OPTION.setRequired(false);
}
@@ -132,11 +132,13 @@ public final class FlinkRunOption {
}
public static Options allOptions() {
+ Options commOptions = getRunCommandOptions();
+ Options yarnOptions = getYARNOptions();
Options resultOptions = new Options();
- for (Option option : getRunCommandOptions().getOptions()) {
+ for (Option option : commOptions.getOptions()) {
resultOptions.addOption(option);
}
- for (Option option : getYarnOptions().getOptions()) {
+ for (Option option : yarnOptions.getOptions()) {
if (!resultOptions.hasOption(option.getOpt())) {
resultOptions.addOption(option);
}
@@ -155,7 +157,7 @@ public final class FlinkRunOption {
return options;
}
- public static Options getYarnOptions() {
+ public static Options getYARNOptions() {
Options allOptions = new Options();
allOptions.addOption(DETACHED_OPTION);
allOptions.addOption(YARN_DETACHED_OPTION);
@@ -188,10 +190,10 @@ public final class FlinkRunOption {
}
public static Options mergeOptions(Options optionsA, Options optionsB) {
+ Options resultOptions = new Options();
if (optionsA == null || optionsB == null) {
throw new IllegalArgumentException("options must not be null");
}
- Options resultOptions = new Options();
for (Option option : optionsA.getOptions()) {
resultOptions.addOption(option);
}
@@ -201,7 +203,12 @@ public final class FlinkRunOption {
return resultOptions;
}
- public static CommandLine parse(Options options, String[] args, boolean stopAtNonOptions) throws ParseException {
- return new DefaultParser().parse(options, args, stopAtNonOptions);
+ public static CommandLine parse(Options options, String[] args, boolean stopAtNonOptions) {
+ DefaultParser parser = new DefaultParser();
+ try {
+ return parser.parse(options, args, stopAtNonOptions);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/ParameterCli.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/ParameterCli.java
index 6b03a2886..5a944d861 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/ParameterCli.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/core/conf/ParameterCli.java
@@ -20,19 +20,19 @@ package org.apache.streampark.flink.core.conf;
import org.apache.streampark.common.conf.ConfigKeys;
import org.apache.streampark.common.util.PropertiesUtils;
-import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.DefaultParser;
import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
import java.io.PrintStream;
import java.net.URLClassLoader;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
+/** Parses Flink application CLI parameters from configuration files. */
public final class ParameterCli {
private static final String PROPERTY_PREFIX = ConfigKeys.KEY_FLINK_PROPERTY_PREFIX();
@@ -45,46 +45,45 @@ public final class ParameterCli {
private ParameterCli() {
}
- public static void main(String[] args) {
- emit(read(args), System.out);
- }
-
- static void emit(String output, PrintStream out) {
+ public static void emit(String output, PrintStream out) {
out.print(output);
}
public static String read(String[] args) {
if ("--vmopt".equals(args[0])) {
- ClassLoader loader = ClassLoader.getSystemClassLoader();
- if (loader instanceof URLClassLoader) {
- return "";
- }
- return "--add-opens java.base/jdk.internal.loader=ALL-UNNAMED "
- + "--add-opens jdk.zipfs/jdk.nio.zipfs=ALL-UNNAMED";
- }
- String action = args[0];
- String conf = args[1];
- Map<String, String> map = loadConfig(conf);
- String[] programArgs = new String[args.length - 2];
- System.arraycopy(args, 2, programArgs, 0, programArgs.length);
+ return readVmOpt();
+ }
+ return readConfigAction(args[0], args[1], Arrays.copyOfRange(args, 2, args.length));
+ }
+
+ private static String readVmOpt() {
+ ClassLoader classLoader = ClassLoader.getSystemClassLoader();
+ if (classLoader instanceof URLClassLoader) {
+ return "";
+ }
+ return "--add-opens java.base/jdk.internal.loader=ALL-UNNAMED "
+ + "--add-opens jdk.zipfs/jdk.nio.zipfs=ALL-UNNAMED";
+ }
+
+ private static String readConfigAction(String action, String conf, String[] programArgs) {
+ Map<String, String> map = loadConfigMap(conf);
switch (action) {
case "--option":
- return buildOption(map, programArgs);
+ return buildOptionString(map, programArgs);
case "--property":
- return buildProperty(map);
+ return buildPropertyString(map);
case "--name":
- return map.getOrDefault(
- PROPERTY_PREFIX + ConfigKeys.KEY_FLINK_APP_NAME(), "").trim();
+ return readAppName(map);
case "--detached":
- return buildDetachedMode(map, programArgs);
+ return readDetachedMode(map, programArgs);
default:
return null;
}
}
- private static Map<String, String> loadConfig(String conf) {
+ private static Map<String, String> loadConfigMap(String conf) {
try {
- String extension = conf.substring(conf.lastIndexOf('.') + 1).toLowerCase(Locale.ROOT);
+ String extension = conf.substring(conf.lastIndexOf('.') + 1).toLowerCase();
switch (extension) {
case "yml":
case "yaml":
@@ -98,24 +97,23 @@ public final class ParameterCli {
"[StreamPark] Usage:flink.conf file error,must be (yml|conf|properties)");
}
} catch (Exception e) {
- throw new IllegalArgumentException(
- "[StreamPark] Failed to load flink config file: " + conf, e);
+ return Collections.emptyMap();
}
}
- private static String buildOption(Map<String, String> map, String[] programArgs) {
+ private static String buildOptionString(Map<String, String> map, String[] programArgs) {
String[] option = getOption(map, programArgs);
StringBuilder buffer = new StringBuilder();
try {
- CommandLine line = PARSER.parse(FLINK_OPTIONS, option, false);
+ org.apache.commons.cli.CommandLine line = PARSER.parse(FLINK_OPTIONS, option, false);
for (org.apache.commons.cli.Option x : line.getOptions()) {
buffer.append(" -").append(x.getOpt());
if (x.hasArg()) {
buffer.append(" ").append(x.getValue());
}
}
- } catch (ParseException exception) {
- // Ignore unrecognized CLI tokens; valid options are still collected below.
+ } catch (Exception exception) {
+ // Ignore invalid CLI options and continue with parsed values.
}
String mainClass = map.get(OPTION_MAIN);
if (mainClass != null) {
@@ -124,74 +122,89 @@ public final class ParameterCli {
return buffer.toString().trim();
}
- private static String buildProperty(Map<String, String> map) {
- StringBuilder buffer = new StringBuilder();
- map.entrySet().stream()
- .filter(
- x -> !OPTION_MAIN.equals(x.getKey())
- && x.getKey().startsWith(PROPERTY_PREFIX)
- && x.getValue() != null
- && !x.getValue().isEmpty())
- .forEach(
- x -> {
- String key = x.getKey().substring(PROPERTY_PREFIX.length()).trim();
- String value = x.getValue().trim();
- if (ConfigKeys.KEY_FLINK_APP_NAME().equals(key)) {
- buffer.append(" -D").append(key).append('=').append(value.replace(' ', '_'));
- } else {
- buffer.append(" -D").append(key).append('=').append(value);
- }
- });
- return buffer.toString().trim();
+ private static String buildPropertyString(Map<String, String> map) {
+ StringBuilder propertyBuffer = new StringBuilder();
+ for (Map.Entry<String, String> entry : map.entrySet()) {
+ appendPropertyEntry(propertyBuffer, entry.getKey(), entry.getValue());
+ }
+ return propertyBuffer.toString().trim();
+ }
+
+ private static void appendPropertyEntry(StringBuilder propertyBuffer, String key, String value) {
+ if (OPTION_MAIN.equals(key)
+ || !key.startsWith(PROPERTY_PREFIX)
+ || value == null
+ || value.isEmpty()) {
+ return;
+ }
+ String propertyKey = key.substring(PROPERTY_PREFIX.length()).trim();
+ String propertyValue = value.trim();
+ propertyBuffer.append(" -D").append(propertyKey).append("=");
+ if (ConfigKeys.KEY_FLINK_APP_NAME().equals(propertyKey)) {
+ propertyBuffer.append(propertyValue.replace(" ", "_"));
+ } else {
+ propertyBuffer.append(propertyValue);
+ }
+ }
+
+ private static String readAppName(Map<String, String> map) {
+ String appName =
+ map.getOrDefault(PROPERTY_PREFIX.concat(ConfigKeys.KEY_FLINK_APP_NAME()), "");
+ appName = appName.trim();
+ return appName.isEmpty() ? "" : appName;
}
- private static String buildDetachedMode(Map<String, String> map, String[] programArgs) {
+ private static String readDetachedMode(Map<String, String> map, String[] programArgs) {
+ String[] detachedOption = getOption(map, programArgs);
try {
- String[] option = getOption(map, programArgs);
- CommandLine line = PARSER.parse(FlinkRunOption.allOptions(), option, false);
+ org.apache.commons.cli.CommandLine line =
+ PARSER.parse(FlinkRunOption.allOptions(), detachedOption, false);
boolean detached =
line.hasOption(FlinkRunOption.DETACHED_OPTION.getOpt())
|| line.hasOption(FlinkRunOption.DETACHED_OPTION.getLongOpt());
return detached ? "Detached" : "Attach";
- } catch (ParseException e) {
- throw new IllegalArgumentException("Failed to parse Flink detached mode options", e);
+ } catch (Exception e) {
+ return "Attach";
}
}
public static String[] getOption(Map<String, String> map, String[] args) {
- Map<String, Object> optionMap = new LinkedHashMap<>();
- mergeConfigOptions(map, optionMap);
- mergeProgramOptions(args, optionMap);
- return flattenOptions(optionMap);
- }
-
- private static void mergeConfigOptions(Map<String, String> map, Map<String, Object> optionMap) {
- map.entrySet().stream()
- .filter(x -> x.getKey().startsWith(OPTION_PREFIX))
- .filter(x -> x.getValue() != null && !x.getValue().isEmpty())
- .filter(
- x -> {
- String key = x.getKey().substring(OPTION_PREFIX.length());
- return FLINK_OPTIONS.hasOption(key);
- })
- .forEach(
- x -> {
- String optKey = "-" + x.getKey().substring(OPTION_PREFIX.length()).trim();
- Object value = parseOptionValue(x.getValue());
- if (value instanceof Boolean && Boolean.TRUE.equals(value)) {
- optionMap.put(optKey, true);
- } else if (!(value instanceof Boolean)) {
- optionMap.put(optKey, value);
- }
- });
- }
-
- private static void mergeProgramOptions(String[] args, Map<String, Object> optionMap) {
+ Map<String, Object> optionMap = collectConfiguredOptions(map);
+ mergeProgramArgs(optionMap, args);
+ return toOptionArray(optionMap);
+ }
+
+ private static Map<String, Object> collectConfiguredOptions(Map<String, String> map) {
+ Map<String, Object> optionMap = new HashMap<>();
+ for (Map.Entry<String, String> entry : map.entrySet()) {
+ putConfiguredOption(optionMap, entry.getKey(), entry.getValue());
+ }
+ return optionMap;
+ }
+
+ private static void putConfiguredOption(Map<String, Object> optionMap, String key, String value) {
+ if (!key.startsWith(OPTION_PREFIX) || value == null || value.isEmpty()) {
+ return;
+ }
+ String optionKey = key.substring(OPTION_PREFIX.length());
+ if (!FLINK_OPTIONS.hasOption(optionKey)) {
+ return;
+ }
+ if ("true".equalsIgnoreCase(value) || "false".equalsIgnoreCase(value)) {
+ if (Boolean.parseBoolean(value)) {
+ optionMap.put("-" + optionKey.trim(), true);
+ }
+ return;
+ }
+ optionMap.put("-" + optionKey.trim(), value);
+ }
+
+ private static void mergeProgramArgs(Map<String, Object> optionMap, String[] args) {
if (args.length == 0) {
return;
}
try {
- CommandLine line = PARSER.parse(FLINK_OPTIONS, args, false);
+ org.apache.commons.cli.CommandLine line = PARSER.parse(FLINK_OPTIONS, args, false);
for (org.apache.commons.cli.Option x : line.getOptions()) {
if (x.hasArg()) {
optionMap.put("-" + x.getLongOpt().trim(), x.getValue());
@@ -199,27 +212,19 @@ public final class ParameterCli {
optionMap.put("-" + x.getLongOpt().trim(), true);
}
}
- } catch (ParseException e) {
- // Ignore unrecognized CLI tokens merged from program arguments.
+ } catch (Exception e) {
+ // Ignore invalid CLI options merged from program args.
}
}
- private static String[] flattenOptions(Map<String, Object> optionMap) {
+ private static String[] toOptionArray(Map<String, Object> optionMap) {
List<String> array = new ArrayList<>();
- optionMap.forEach(
- (key, value) -> {
- array.add(key);
- if (value instanceof String) {
- array.add(value.toString());
- }
- });
- return array.toArray(new String[0]);
- }
-
- private static Object parseOptionValue(String raw) {
- if ("true".equalsIgnoreCase(raw) || "false".equalsIgnoreCase(raw)) {
- return Boolean.parseBoolean(raw);
+ for (Map.Entry<String, Object> entry : optionMap.entrySet()) {
+ array.add(entry.getKey());
+ if (entry.getValue() instanceof String) {
+ array.add(entry.getValue().toString());
+ }
}
- return raw;
+ return array.toArray(new String[0]);
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
index 2df697fa8..6bf7d9ae3 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-base/src/main/java/org/apache/streampark/flink/util/FlinkUtils.java
@@ -17,18 +17,39 @@
package org.apache.streampark.flink.util;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
import org.apache.flink.util.TimeUtils;
import java.io.File;
import java.time.Duration;
+import java.util.Arrays;
import java.util.Map;
+import java.util.stream.Collectors;
+/** Flink utility methods. */
public final class FlinkUtils {
private FlinkUtils() {
}
+ public static <R> ListState<R> getUnionListState(
+ FunctionInitializationContext context,
+ String descriptorName,
+ TypeInformation<R> typeInformation) {
+ try {
+ return context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ descriptorName, typeInformation.getTypeClass()));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
public static String getFlinkDistJar(String flinkHome) {
String[] jars =
new File(flinkHome + "/lib")
@@ -44,14 +65,15 @@ public final class FlinkUtils {
"[StreamPark] found multiple flink-dist jar in "
+ flinkHome
+ "/lib,["
- + String.join(",", jars)
+ + Arrays.stream(jars).collect(Collectors.joining(","))
+ "]");
}
public static boolean isCheckpointEnabled(Map<String, String> map) {
Duration checkpointInterval =
TimeUtils.parseDuration(
- map.getOrDefault(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL.key(), "0ms"));
+ map.getOrDefault(
+ ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL.key(), "0ms"));
return checkpointInterval.toMillis() > 0;
}
}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims-test/pom.xml b/streampark-flink/streampark-flink-shims/streampark-flink-shims-test/pom.xml
index 2bd7d9c3a..cf6f2be43 100644
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims-test/pom.xml
+++ b/streampark-flink/streampark-flink-shims/streampark-flink-shims-test/pom.xml
@@ -25,33 +25,32 @@
<version>${revision}</version>
</parent>
- <artifactId>streampark-flink-shims-test_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-test</artifactId>
<name>StreamPark : Flink Shims Test</name>
<properties>
- <flink.version>1.20.1</flink.version>
+ <!-- Override both when testing another Flink version, e.g. -Dstreampark.flink.shims.version=1.20 -Dflink.version=1.20.1 -->
+ <streampark.flink.shims.version>1.17</streampark.flink.shims.version>
+ <flink.version>1.17.0</flink.version>
</properties>
<dependencies>
-
<dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- <scope>test</scope>
+ <groupId>org.apache.streampark</groupId>
+ <artifactId>streampark-common</artifactId>
+ <version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims-base_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims-base</artifactId>
<version>${project.version}</version>
- <scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.streampark</groupId>
- <artifactId>streampark-flink-shims_flink-1.14_${scala.binary.version}</artifactId>
+ <artifactId>streampark-flink-shims_flink-${streampark.flink.shims.version}</artifactId>
<version>${project.version}</version>
- <scope>test</scope>
</dependency>
<dependency>
@@ -63,32 +62,11 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-scala_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <optional>true</optional>
- </dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
@@ -98,18 +76,16 @@
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-statebackend-rocksdb</artifactId>
- <version>${flink.version}</version>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
-
</dependencies>
<profiles>
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java
deleted file mode 100644
index 29a609346..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.api.java.utils.ParameterTool;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment;
-import org.apache.flink.table.api.StatementSet;
-import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
-import org.apache.flink.table.descriptors.ConnectorDescriptor;
-import org.apache.flink.table.descriptors.StreamTableDescriptor;
-import org.apache.flink.table.sources.TableSource;
-
-import scala.Tuple3;
-
-/** Integration api of stream and table */
-public class StreamTableContext extends FlinkStreamTableTrait {
-
- public StreamTableContext(
- ParameterTool parameter,
- StreamExecutionEnvironment streamEnv,
- StreamTableEnvironment tableEnv) {
- super(parameter, streamEnv, tableEnv);
- }
-
- public StreamTableContext(
- Tuple3<ParameterTool, StreamExecutionEnvironment, StreamTableEnvironment> args) {
- this(args._1(), args._2(), args._3());
- }
-
- public StreamTableContext(StreamTableEnvConfig args) {
- this(FlinkTableInitializer.initialize(args));
- }
-
- @Override
- public StreamTableDescriptor connect(ConnectorDescriptor connectorDescriptor) {
- return tableEnv().connect(connectorDescriptor);
- }
-
- public StreamGraph $getStreamGraph(String jobName) {
- return streamEnv().getStreamGraph(jobName);
- }
-
- public StreamGraph $getStreamGraph(String jobName, boolean clearTransformations) {
- return streamEnv().getStreamGraph(jobName, clearTransformations);
- }
-
- @Override
- public StatementSet createStatementSet() {
- return tableEnv().createStatementSet();
- }
-
- @Override
- public Table fromTableSource(TableSource<?> source) {
- return tableEnv().fromTableSource(source);
- }
-
- @Override
- public void insertInto(Table table, String sinkPath, String... sinkPathContinued) {
- tableEnv().insertInto(table, sinkPath, sinkPathContinued);
- }
-
- @Override
- public void insertInto(String targetPath, Table table) {
- tableEnv().insertInto(targetPath, table);
- }
-
- @Override
- public String explain(Table table) {
- return tableEnv().explain(table);
- }
-
- @Override
- public String explain(Table table, boolean extended) {
- return tableEnv().explain(table, extended);
- }
-
- @Override
- public String explain(boolean extended) {
- return tableEnv().explain(extended);
- }
-
- @Override
- public void sqlUpdate(String stmt) {
- tableEnv().sqlUpdate(stmt);
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableContext.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableContext.java
deleted file mode 100644
index 8aebb0f92..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableContext.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.api.common.JobExecutionResult;
-import org.apache.flink.api.java.utils.ParameterTool;
-import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.table.descriptors.ConnectTableDescriptor;
-import org.apache.flink.table.descriptors.ConnectorDescriptor;
-import org.apache.flink.table.sources.TableSource;
-
-import scala.Tuple2;
-
-public class TableContext extends FlinkTableTrait {
-
- public TableContext(ParameterTool parameter, TableEnvironment tableEnv) {
- super(parameter, tableEnv);
- }
-
- public TableContext(Tuple2<ParameterTool, TableEnvironment> args) {
- this(args._1(), args._2());
- }
-
- public TableContext(TableEnvConfig args) {
- this(FlinkTableInitializer.initialize(args));
- }
-
- @Override
- public ConnectTableDescriptor connect(ConnectorDescriptor connectorDescriptor) {
- return delegate().connect(connectorDescriptor);
- }
-
- @Override
- public JobExecutionResult execute(String jobName) {
- return printStartupLogo(jobName);
- }
-
- @Override
- public Table fromTableSource(TableSource<?> source) {
- return delegate().fromTableSource(source);
- }
-
- @Override
- public void insertInto(Table table, String sinkPath, String... sinkPathContinued) {
- delegate().insertInto(table, sinkPath, sinkPathContinued);
- }
-
- @Override
- public void insertInto(String targetPath, Table table) {
- delegate().insertInto(targetPath, table);
- }
-
- @Override
- public String explain(Table table) {
- return delegate().explain(table);
- }
-
- @Override
- public String explain(Table table, boolean extended) {
- return delegate().explain(table, extended);
- }
-
- @Override
- public String explain(boolean extended) {
- return delegate().explain(extended);
- }
-
- @Override
- public void sqlUpdate(String stmt) {
- delegate().sqlUpdate(stmt);
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableExt.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableExt.java
deleted file mode 100644
index 61f029ef2..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.12/src/main/java/org/apache/streampark/flink/core/TableExt.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.streaming.api.scala.DataStream;
-import org.apache.flink.table.api.Table;
-
-/**
- * Table extension utilities for Flink Table API.
- */
-
-public final class TableExt {
-
- private TableExt() {
- }
-
- public static TableWrapper wrap(Table table) {
- return new TableWrapper(table);
- }
-
- public static TableConversions conversions(Table table) {
- return new TableConversions(table);
- }
-
- public static final class TableWrapper {
-
- private final Table table;
-
- public TableWrapper(Table table) {
- this.table = table;
- }
-
- public Table alias(String field, String... fields) {
- return table.as(field, fields);
- }
-
- }
-
- public static final class TableConversions extends org.apache.flink.table.api.bridge.scala.TableConversions {
-
- public <T> org.apache.flink.api.scala.DataSet<T> toDataSet(TypeInformation<T> typeInfo) {
- return super.toDataSet(typeInfo);
- }
-
- public <T> DataStream<T> appendStream(StreamTableContext context, TypeInformation<T> typeInfo) {
- context.isConvertedToDataStream = true;
- return super.toAppendStream(typeInfo);
- }
-
- public <T> DataStream<scala.Tuple2<Object, T>> retractStream(
- StreamTableContext context,
- TypeInformation<T> typeInfo) {
- context.isConvertedToDataStream = true;
- return super.toRetractStream(typeInfo);
- }
-
- public <T> DataStream<T> toAppendStream(StreamTableContext context, TypeInformation<T> typeInfo) {
- return appendStream(context, typeInfo);
- }
-
- public <T> DataStream<scala.Tuple2<Object, T>> toRetractStream(
- StreamTableContext context,
- TypeInformation<T> typeInfo) {
- return retractStream(context, typeInfo);
- }
-
- public TableConversions(Table table) {
- super(table);
- }
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
deleted file mode 100644
index 4914c46e9..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkClusterClient.java
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.client.program.ClusterClient;
-
-public class FlinkClusterClient<T> extends FlinkClientTrait<T> {
-
- public FlinkClusterClient(ClusterClient<T> clusterClient) {
- super(clusterClient);
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java
deleted file mode 100644
index e300de2a6..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/FlinkKubernetesClient.java
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
-
-public class FlinkKubernetesClient extends FlinkKubernetesClientTrait {
-
- public FlinkKubernetesClient(FlinkKubeClient kubeClient) {
- super(kubeClient);
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java
deleted file mode 100644
index 29a609346..000000000
--- a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/StreamTableContext.java
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.streampark.flink.core;
-
-import org.apache.flink.api.java.utils.ParameterTool;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment;
-import org.apache.flink.table.api.StatementSet;
-import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment;
-import org.apache.flink.table.descriptors.ConnectorDescriptor;
-import org.apache.flink.table.descriptors.StreamTableDescriptor;
-import org.apache.flink.table.sources.TableSource;
-
-import scala.Tuple3;
-
-/** Integration api of stream and table */
-public class StreamTableContext extends FlinkStreamTableTrait {
-
- public StreamTableContext(
- ParameterTool parameter,
- StreamExecutionEnvironment streamEnv,
- StreamTableEnvironment tableEnv) {
- super(parameter, streamEnv, tableEnv);
- }
-
- public StreamTableContext(
- Tuple3<ParameterTool, StreamExecutionEnvironment, StreamTableEnvironment> args) {
- this(args._1(), args._2(), args._3());
- }
-
- public StreamTableContext(StreamTableEnvConfig args) {
- this(FlinkTableInitializer.initialize(args));
- }
-
- @Override
- public StreamTableDescriptor connect(ConnectorDescriptor connectorDescriptor) {
- return tableEnv().connect(connectorDescriptor);
- }
-
- public StreamGraph $getStreamGraph(String jobName) {
- return streamEnv().getStreamGraph(jobName);
- }
-
- public StreamGraph $getStreamGraph(String jobName, boolean clearTransformations) {
- return streamEnv().getStreamGraph(jobName, clearTransformations);
- }
-
- @Override
- public StatementSet createStatementSet() {
- return tableEnv().createStatementSet();
- }
-
- @Override
- public Table fromTableSource(TableSource<?> source) {
- return tableEnv().fromTableSource(source);
- }
-
- @Override
- public void insertInto(Table table, String sinkPath, String... sinkPathContinued) {
- tableEnv().insertInto(table, sinkPath, sinkPathContinued);
- }
-
- @Override
- public void insertInto(String targetPath, Table table) {
- tableEnv().insertInto(targetPath, table);
- }
-
- @Override
- public String explain(Table table) {
- return tableEnv().explain(table);
- }
-
- @Override
- public String explain(Table table, boolean extended) {
- return tableEnv().explain(table, extended);
- }
-
- @Override
- public String explain(boolean extended) {
- return tableEnv().explain(extended);
- }
-
- @Override
- public void sqlUpdate(String stmt) {
- tableEnv().sqlUpdate(stmt);
- }
-}
diff --git a/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/TableContext.java b/streampark-flink/streampark-flink-shims/streampark-flink-shims_flink-1.13/src/main/java/org/apache/streampark/flink/core/TableContext.java
... 5854 lines suppressed ...