You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "macdoor (via GitHub)" <gi...@apache.org> on 2026/10/10 08:17:21 UTC
[PR] [Feature-18691][task-flink] Submit Flink SQL through SQL Gateway [dolphinscheduler]
macdoor opened a new pull request, #18693:
URL: https://github.com/apache/dolphinscheduler/pull/18693
## Was this PR generated or assisted by AI?
**YES** — the SQL Gateway submit path, UI fields, and unit tests were assisted by an AI agent (Hermes). The change was human-reviewed. Unit tests passed, and a FLINK task with `sqlSubmitType=SQL_GATEWAY` was smoked on an internal cluster.
## Purpose of the pull request
Add an optional SQL Gateway submit mode to the existing **FLINK** task, so batch SQL can be sent to a remote Flink SQL Gateway over JDBC without a local Flink client.
This follows the review on #18690: do not add a new task type. The existing `sql-client.sh` path stays the default. Closes #18691.
## Brief change log
- `FlinkParameters.sqlSubmitType`: absent / `CLIENT` keeps `${FLINK_HOME}/bin/sql-client.sh`; `SQL_GATEWAY` submits over JDBC and cancels via `statement.cancel()`
- Fork only at `FlinkTask.handle()` / `cancelApplication()`. `getScript()` and `FlinkArgsUtils` are unchanged
- Script sources reuse `SqlSourceType`. The JDBC driver is shaded only into `dolphinscheduler-task-flink`; `dolphinscheduler-task-flink-stream` excludes it
- UI shows the Gateway fields only when program type is SQL and submit type is SQL Gateway
## Verify this pull request
This change added tests and can be verified as follows:
- `mvn test -pl dolphinscheduler-task-plugin/dolphinscheduler-task-flink -Dtest=FlinkParametersTest,FlinkSqlGatewayExecutorTest,FlinkArgsUtilsTest,FlinkTaskTest -Dspotless.skip=true -Djacoco.skip=true`
- `mvn -pl dolphinscheduler-task-plugin/dolphinscheduler-task-flink,dolphinscheduler-task-plugin/dolphinscheduler-task-flink-stream spotless:check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-18691][task-flink] Submit Flink SQL through SQL Gateway [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed pull request #18693: [Feature-18691][task-flink] Submit Flink SQL through SQL Gateway
URL: https://github.com/apache/dolphinscheduler/pull/18693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-18691][task-flink] Submit Flink SQL through SQL Gateway [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on PR #18693:
URL: https://github.com/apache/dolphinscheduler/pull/18693#issuecomment-6463283022
We have no plans to abandon jdk8 for now. In order to introduce this relatively less core component, we have made such a huge change. I'm -1 on this. Closing this PR for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-18691][task-flink] Submit Flink SQL through SQL Gateway [dolphinscheduler]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor commented on PR #18693:
URL: https://github.com/apache/dolphinscheduler/pull/18693#issuecomment-6118437528
Addressed the review on ea309e3d66.
1. The executor adds `SET 'table.dml-sync'='true'` itself, before the init script and the main script. This is not a Gateway default and is not left to user SQL. With that flag, DML blocks until the job reaches a terminal state. A later job failure is thrown by the driver and the task is marked FAILURE, not SUCCESS. We do not poll the JobManager.
2. A kill during `getConnection()` is published under a lock and is not overwritten with SUCCESS.
3. Statement splitting skips separators inside quotes, escaped quotes, and `--` / `/* */` comments. Covered by unit tests.
4. `SQL_GATEWAY` is accepted only when `programType` is SQL. Leaving SQL serializes `sqlSubmitType` as `CLIENT`.
5. Gateway controls are shown only for `taskType=FLINK`, not `FLINK_STREAM`.
6. `flink-sql-jdbc-driver-bundle` 2.2.0 is Java 11 bytecode. `SQL_GATEWAY` fails in init on JDK 8 and states that JDK 11+ is required. The official bundle is provided on the classpath and is not shaded into the plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org