You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "platinumhamburg (via GitHub)" <gi...@apache.org> on 2026/05/29 06:54:38 UTC
[I] [flink-action][server][client] add orphan files cleanup action for remote storage [fluss]
platinumhamburg opened a new issue, #3403:
URL: https://github.com/apache/fluss/issues/3403
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Fluss persists log segments and KV snapshots to remote storage (S3, OSS, HDFS, etc.). Over time, orphan files accumulate — files that consume storage costs but are no longer referenced by any live metadata. These originate from:
1. **Failed commits.** Files written to remote storage but never registered in coordinator metadata (process crash, network partition).
2. **Interrupted deletions.** Metadata removed but corresponding remote files not yet cleaned up due to transient failures or leader failover.
3. **Superseded manifests.** Remote log manifests use upsert semantic — old manifest files are left behind after replacement.
Without a dedicated cleanup mechanism, orphan files grow monotonically. In production clusters with high write throughput, this can reach terabytes of wasted capacity within weeks.
### Solution
Introduce a Flink-based `orphan_files_clean` action that safely identifies and deletes orphan remote files via a 3-stage DAG:
- **ScopeEnumerator (p=1):** Queries coordinator via two new read-only RPCs (`ListRemoteLogManifests`, `ListKvSnapshots`) to build the active reference set. Detects orphan table/partition directories via ID guards. Emits per-bucket `CleanTask` items.
- **ScanAndClean (p=N):** Walks log/kv directories on remote storage, applies rule-based file classification against the active set, and deletes orphan files with rate limiting.
- **StatsAggregate (p=1):** Collects cleanup stats and performs a final empty-directory sweep.
**CLI usage:**
```bash
bin/flink run fluss-flink-action.jar orphan_files_clean \
--bootstrap-server host:port \
--database mydb \
--dry-run \
--older-than "2025-05-26 00:00:00" \
--delete-rate-limit-per-second 100
```
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [flink-action][server][client] add orphan files cleanup action for remote storage [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on issue #3403:
URL: https://github.com/apache/fluss/issues/3403#issuecomment-4582873378
@platinumhamburg
Hi, I have a suggestion.
Is it possible to introduce a usage method based on "Flink procedures," similar to that provided by Paimon:
https://paimon.apache.org/docs/1.4/flink/procedures/
like:
```sql
CALL sys. orphan_files_clean(`database ` => 'mydb', older_than => '2023-10-31 12:00:00')
```
It is more suitable for development targeting FlinkSQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [flink-action][server][client] add orphan files cleanup action for remote storage [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on issue #3403:
URL: https://github.com/apache/fluss/issues/3403#issuecomment-4598188965
> Hi, I have a suggestion. Is it possible to introduce a usage method based on "Flink procedures," similar to that provided by Paimon: https://paimon.apache.org/docs/1.4/flink/procedures/ like:
>
> CALL sys. orphan_files_clean(`database ` => 'mydb', older_than => '2023-10-31 12:00:00')
> It is more suitable for development targeting FlinkSQL.
Hi @sd4324530 Thank you for the suggestion. In fact, we have discussed this idea before. However, in the case of Fluss, orphan files are usually a low-probability problem caused by abnormal cluster scenarios. The responsibility for cleaning them mostly falls on the cluster maintainer rather than the user. This is exactly the opposite of Paimon, where users themselves are largely responsible for orphan file cleanup. Therefore, we’ve decided not to provide a procedure entry for now, and we will add it later if it becomes truly necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [flink-action][server][client] add orphan files cleanup action for remote storage [fluss]
Posted by "sd4324530 (via GitHub)" <gi...@apache.org>.
sd4324530 commented on issue #3403:
URL: https://github.com/apache/fluss/issues/3403#issuecomment-4598523592
@platinumhamburg I understand, thank you very much for your explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org