You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yunfengzhou-hub (via GitHub)" <gi...@apache.org> on 2025/09/05 01:31:23 UTC
[PR] [flink] Support starting Flink job for all actions [paimon]
yunfengzhou-hub opened a new pull request, #6201:
URL: https://github.com/apache/paimon/pull/6201
### Purpose
Currently Flink actions do not provide a unified experience for users. Some actions like CompactAction will start a Flink job, but others like CreateTagAction are lightweight and will not start a Flink job. This brings more complexity when DevOps are trying to integrate Flink Actions to their platforms, because they need to provide multiple execution pipelines for different Actions.
In order to solve this problem, this PR proposes to introduce the parameter `force_start_flink_job` and the method `ActionBase#forceStartFlinkJob` to support all actions to execute within a Flink job. This PR mainly includes the following changes:
* Adds `LocalAction` interface and uses it to mark actions that does not natively need a flink job.
* Makes `LocalAction` serializable, as a prerequisite to wrap them into Flink operators.
* Adds a common implementation in `ActionBase` to execute `LocalAction`s in Flink job.
* Introduces the parameter `force_start_flink_job` and the method `ActionBase#forceStartFlinkJob` as user interface.
### Tests
* Existing action unit tests are modified to test the flink job mode, for example `BranchActionITCase`.
* `ActionJobCoverageTest` is introduced to make sure all actions have provided a Flink job execution mode. This test case will also remind developers to support this feature when new actions are introduced in future.
### API and Format
This PR introduces the parameter `force_start_flink_job` and the method `ActionBase#forceStartFlinkJob` to support all actions to execute within a Flink job.
### Documentation
The new feature is documented in the Flink Actions section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support starting Flink job for all actions [paimon]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #6201:
URL: https://github.com/apache/paimon/pull/6201#issuecomment-3256846380
Hi @Sxnan could you please help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support starting Flink job for all actions [paimon]
Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin merged PR #6201:
URL: https://github.com/apache/paimon/pull/6201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support starting Flink job for all actions [paimon]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #6201:
URL: https://github.com/apache/paimon/pull/6201#issuecomment-3302205596
Hi @yuzelin could you please help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support starting Flink job for all actions [paimon]
Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on code in PR #6201:
URL: https://github.com/apache/paimon/pull/6201#discussion_r2371506439
##########
docs/content/flink/action-jars.md:
##########
@@ -305,3 +305,21 @@ For more information of 'rewrite_file_index', see
/path/to/paimon-flink-action-{{< version >}}.jar \
rewrite_file_index --help
```
+
+## Force Start Flink Job
+
+Some actions, like `create_tag`, are lightweight and by default will not be submitted as a job to Flink cluster. If you
+have the need to unify the experience regardless of actions, you can use the `--force_start_flink_job` flag to make sure
+submitting them as jobs. For example,
+
+```bash
+<FLINK_HOME>/bin/flink run \
+ /path/to/paimon-flink-action-{{< version >}}.jar \
+ drop_partition \
+ --warehouse <warehouse-path> \
+ --database <database-name> \
+ --table <table-name> \
+ [--partition <partition_spec> [--partition <partition_spec> ...]] \
+ [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
+ --force_start_flink_job
Review Comment:
How about `--force_start_flink_job true/false (default false)`? It's more similar to the style of other arguments.
And it's better to use method `forceStartFlinkJob(boolean);` so you can create test codes like `action.forceStartFlinkJob(forceStartFlinkJob);` instead of an `if` statement.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/CompactAction.java:
##########
@@ -282,7 +283,12 @@ private boolean buildForPostponeBucketCompaction(
.withBucket(BucketMode.POSTPONE_BUCKET)
.partitions();
if (partitions.isEmpty()) {
- return false;
+ if (this.forceStartFlinkJob) {
+ env.fromSequence(0, 0).name("Placeholder Source").sinkTo(new DiscardingSink<>());
Review Comment:
I think 'Nothing to Compact Source' is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org