You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhuyaogai (via GitHub)" <gi...@apache.org> on 2026/08/16 06:11:12 UTC
[PR] [flink] Support table batching for orphan file cleanup [paimon]
zhuyaogai opened a new pull request, #9247:
URL: https://github.com/apache/paimon/pull/9247
### Purpose
Fixes #7155.
Previously, orphan file cleanup for all matched tables was unioned into a single Flink batch job. In batch mode, Flink does not put all job vertices into the same SlotSharingGroup by default, so tasks belonging to different groups cannot reuse the same slots.
As each table creates its own cleanup pipeline, running many tables in one job can require a large number of slots. Therefore, the actual resource usage may be much higher than the configured cleanup parallelism and can exhaust cluster resources.
This change processes tables in sequential batches and introduces a `table_batch_size` option, which defaults to 10. Only a limited number of table cleanup pipelines are included in each batch, bounding the slot resources required by a single job. Batch progress and elapsed time are also logged.
### Tests
Added integration test coverage for database-level orphan file cleanup with a configured batch size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #9247:
URL: https://github.com/apache/paimon/pull/9247#discussion_r3813936166
##########
docs/docs/flink/procedures.md:
##########
@@ -407,13 +407,13 @@ All available procedures are listed below.
<td>remove_orphan_files</td>
<td>
-- Use named argument<br/>
- CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode') <br/><br/>
+ CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode', table_batch_size => 'tableBatchSize') <br/><br/>
Review Comment:
Hi @JingsongLi, could you elaborate a bit on what you have in mind with `max_table_number`?
The current design submits all matched tables in sequential Flink jobs, and this parameter controls how many tables are included in each job.
I'm not sure whether your suggestion is only about the parameter name, or whether you have different semantics or an alternative design in mind. I'd like to understand your intention before changing the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #9247:
URL: https://github.com/apache/paimon/pull/9247#discussion_r3823768467
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -436,40 +438,85 @@ public static CleanOrphanFilesResult executeDatabaseOrphanFiles(
String databaseName,
@Nullable String tableName)
throws Catalog.DatabaseNotExistException, Catalog.TableNotExistException {
+ return executeDatabaseOrphanFiles(
+ env,
+ catalog,
+ olderThanMillis,
+ dryRun,
+ parallelism,
+ databaseName,
+ tableName,
+ DEFAULT_TABLE_BATCH_SIZE);
+ }
+
+ public static CleanOrphanFilesResult executeDatabaseOrphanFiles(
+ StreamExecutionEnvironment env,
+ Catalog catalog,
+ long olderThanMillis,
+ boolean dryRun,
+ @Nullable Integer parallelism,
+ String databaseName,
+ @Nullable String tableName,
+ @Nullable Integer tableBatchSize)
+ throws Catalog.DatabaseNotExistException, Catalog.TableNotExistException {
List<String> tableNames = Collections.singletonList(tableName);
if (tableName == null || "*".equals(tableName)) {
tableNames = catalog.listTables(databaseName);
}
- List<DataStream<CleanOrphanFilesResult>> orphanFilesCleans =
- new ArrayList<>(tableNames.size());
- for (String t : tableNames) {
- Identifier identifier = new Identifier(databaseName, t);
- Table table = catalog.getTable(identifier);
- checkArgument(
- table instanceof FileStoreTable,
- "Only FileStoreTable supports remove-orphan-files action. The table type is '%s'.",
- table.getClass().getName());
-
- DataStream<CleanOrphanFilesResult> clean =
- new FlinkOrphanFilesClean(
- (FileStoreTable) table, olderThanMillis, dryRun, parallelism)
- .doOrphanClean(env);
- if (clean != null) {
- orphanFilesCleans.add(clean);
+ int batchSize = tableBatchSize == null ? DEFAULT_TABLE_BATCH_SIZE : tableBatchSize;
+ checkArgument(batchSize > 0, "Table batch size must be greater than 0.");
+
+ long deletedFilesCount = 0;
+ long deletedFilesLenInBytes = 0;
+ for (int start = 0; start < tableNames.size(); start += batchSize) {
+ int end = Math.min(start + batchSize, tableNames.size());
+ int batchNumber = start / batchSize + 1;
+ int tableCount = end - start;
+ long batchStart = System.currentTimeMillis();
+ LOG.info(
+ "Starting orphan files clean batch #{} with {} tables.",
+ batchNumber,
+ tableCount);
+
+ List<DataStream<CleanOrphanFilesResult>> orphanFilesCleans =
+ new ArrayList<>(tableCount);
+ for (String t : tableNames.subList(start, end)) {
+ Identifier identifier = new Identifier(databaseName, t);
+ Table table = catalog.getTable(identifier);
+ checkArgument(
+ table instanceof FileStoreTable,
+ "Only FileStoreTable supports remove-orphan-files action. The table type is '%s'.",
+ table.getClass().getName());
+
+ DataStream<CleanOrphanFilesResult> clean =
+ new FlinkOrphanFilesClean(
+ (FileStoreTable) table,
+ olderThanMillis,
+ dryRun,
+ parallelism)
+ .doOrphanClean(env);
+ if (clean != null) {
+ orphanFilesCleans.add(clean);
+ }
}
- }
- DataStream<CleanOrphanFilesResult> result = null;
- for (DataStream<CleanOrphanFilesResult> clean : orphanFilesCleans) {
- if (result == null) {
- result = clean;
- } else {
- result = result.union(clean);
+ DataStream<CleanOrphanFilesResult> result = null;
+ for (DataStream<CleanOrphanFilesResult> clean : orphanFilesCleans) {
+ result = result == null ? clean : result.union(clean);
Review Comment:
Yes, tables within one batch are added to the same Flink JobGraph through `union`, so their cleanup pipelines may execute concurrently.
Different batches are executed sequentially. `sum(result)` calls `executeAndCollect()` and consumes the iterator until the current Flink job finishes. Only then does the outer loop build and submit the next job.
Therefore, the parameter limits the number of tables included in each submitted Flink job, rather than directly limiting task concurrency. Does this execution model match what you had in mind, or would you prefer a different way to control resource usage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9247:
URL: https://github.com/apache/paimon/pull/9247#discussion_r3793786152
##########
docs/docs/flink/procedures.md:
##########
@@ -407,13 +407,13 @@ All available procedures are listed below.
<td>remove_orphan_files</td>
<td>
-- Use named argument<br/>
- CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode') <br/><br/>
+ CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode', table_batch_size => 'tableBatchSize') <br/><br/>
Review Comment:
Maybe max_table_number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9247:
URL: https://github.com/apache/paimon/pull/9247#issuecomment-5466140925
I am concerned about whether various Flink runtime environments support operations like this—specifically, the iterative processing at which Spark excels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #9247:
URL: https://github.com/apache/paimon/pull/9247#discussion_r3797373502
##########
docs/docs/flink/procedures.md:
##########
@@ -407,13 +407,13 @@ All available procedures are listed below.
<td>remove_orphan_files</td>
<td>
-- Use named argument<br/>
- CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode') <br/><br/>
+ CALL [catalog.]sys.remove_orphan_files(`table` => 'identifier', older_than => 'olderThan', dry_run => 'dryRun', mode => 'mode', table_batch_size => 'tableBatchSize') <br/><br/>
Review Comment:
This parameter controls the number of tables included in each batch (each submitted Flink job), rather than limiting the total number of tables to clean.
For example, 23 tables with a value of 10 will be processed in three batches: 10, 10, and 3. All 23 tables will still be cleaned.
`max_table_number` might be interpreted as a limit on the total number of tables to process. I think `table_batch_size` better reflects the current semantics. Would `max_tables_per_batch` be clearer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9247:
URL: https://github.com/apache/paimon/pull/9247#discussion_r3818156848
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -436,40 +438,85 @@ public static CleanOrphanFilesResult executeDatabaseOrphanFiles(
String databaseName,
@Nullable String tableName)
throws Catalog.DatabaseNotExistException, Catalog.TableNotExistException {
+ return executeDatabaseOrphanFiles(
+ env,
+ catalog,
+ olderThanMillis,
+ dryRun,
+ parallelism,
+ databaseName,
+ tableName,
+ DEFAULT_TABLE_BATCH_SIZE);
+ }
+
+ public static CleanOrphanFilesResult executeDatabaseOrphanFiles(
+ StreamExecutionEnvironment env,
+ Catalog catalog,
+ long olderThanMillis,
+ boolean dryRun,
+ @Nullable Integer parallelism,
+ String databaseName,
+ @Nullable String tableName,
+ @Nullable Integer tableBatchSize)
+ throws Catalog.DatabaseNotExistException, Catalog.TableNotExistException {
List<String> tableNames = Collections.singletonList(tableName);
if (tableName == null || "*".equals(tableName)) {
tableNames = catalog.listTables(databaseName);
}
- List<DataStream<CleanOrphanFilesResult>> orphanFilesCleans =
- new ArrayList<>(tableNames.size());
- for (String t : tableNames) {
- Identifier identifier = new Identifier(databaseName, t);
- Table table = catalog.getTable(identifier);
- checkArgument(
- table instanceof FileStoreTable,
- "Only FileStoreTable supports remove-orphan-files action. The table type is '%s'.",
- table.getClass().getName());
-
- DataStream<CleanOrphanFilesResult> clean =
- new FlinkOrphanFilesClean(
- (FileStoreTable) table, olderThanMillis, dryRun, parallelism)
- .doOrphanClean(env);
- if (clean != null) {
- orphanFilesCleans.add(clean);
+ int batchSize = tableBatchSize == null ? DEFAULT_TABLE_BATCH_SIZE : tableBatchSize;
+ checkArgument(batchSize > 0, "Table batch size must be greater than 0.");
+
+ long deletedFilesCount = 0;
+ long deletedFilesLenInBytes = 0;
+ for (int start = 0; start < tableNames.size(); start += batchSize) {
+ int end = Math.min(start + batchSize, tableNames.size());
+ int batchNumber = start / batchSize + 1;
+ int tableCount = end - start;
+ long batchStart = System.currentTimeMillis();
+ LOG.info(
+ "Starting orphan files clean batch #{} with {} tables.",
+ batchNumber,
+ tableCount);
+
+ List<DataStream<CleanOrphanFilesResult>> orphanFilesCleans =
+ new ArrayList<>(tableCount);
+ for (String t : tableNames.subList(start, end)) {
+ Identifier identifier = new Identifier(databaseName, t);
+ Table table = catalog.getTable(identifier);
+ checkArgument(
+ table instanceof FileStoreTable,
+ "Only FileStoreTable supports remove-orphan-files action. The table type is '%s'.",
+ table.getClass().getName());
+
+ DataStream<CleanOrphanFilesResult> clean =
+ new FlinkOrphanFilesClean(
+ (FileStoreTable) table,
+ olderThanMillis,
+ dryRun,
+ parallelism)
+ .doOrphanClean(env);
+ if (clean != null) {
+ orphanFilesCleans.add(clean);
+ }
}
- }
- DataStream<CleanOrphanFilesResult> result = null;
- for (DataStream<CleanOrphanFilesResult> clean : orphanFilesCleans) {
- if (result == null) {
- result = clean;
- } else {
- result = result.union(clean);
+ DataStream<CleanOrphanFilesResult> result = null;
+ for (DataStream<CleanOrphanFilesResult> clean : orphanFilesCleans) {
+ result = result == null ? clean : result.union(clean);
Review Comment:
I don't quite understand the execution process here—is it concurrent execution? Why is it described in terms of batches?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support table batching for orphan file cleanup [paimon]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on PR #9247:
URL: https://github.com/apache/paimon/pull/9247#issuecomment-6312239255
Hi, is this still in progress?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org