You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sundapeng (via GitHub)" <gi...@apache.org> on 2026/08/26 05:46:27 UTC
[PR] [common][oss] Add a strict batch delete capability to FileIO [paimon]
sundapeng opened a new pull request, #9398:
URL: https://github.com/apache/paimon/pull/9398
### Purpose
Deleting objects one request at a time is what dominates a format table overwrite commit on object
storage, and OSS can delete up to 1000 keys of one bucket in a single request. This adds the
capability and an OSS provider. The Format Table side that consumes it is a separate change.
`FileIO.batchFileDeleter(Path)` returns an optional batch deleter for the provider serving that path.
The default is empty and performs no storage access, so existing implementations keep working
unchanged, and `PluginFileIO`, `ResolvingFileIO`, `CachingFileIO` and `RESTTokenFileIO` forward it.
The contract is strict. An absent capability is the only signal that a caller may use individual
deletes; once a batch request has started, a failure or an incomplete response fails the caller
rather than deleting the remaining files one by one, which would hide a partial success. A caller
that retries must resend the same complete batch. An implementation validates the whole request,
bucket and authority included, before it touches a client.
`OSSFileIO` implements it over `deleteObjects` with at most 1000 keys of one bucket per request, and
verifies the per-object result against what was asked for.
Jindo is the other provider that matters for this workload and it is not here: 6.9.1 has no batch
API, and while 6.10.7 has one it is disabled by default with single failover on, so it needs its own
change backed by runtime evidence.
### Tests
`FileIOBatchDeleteContractTest` and `FileIOBatchDeleteForwardingTest` for the capability and the four
wrappers, including discovery that performs no I/O, a plugin invocation restoring the context
classloader, mixed authorities under `ResolvingFileIO`, rediscovery after a REST token refresh, and
an immutable result.
`OSSFileIOBatchDeleteTest` for the provider: batch size, bucket and authority validation before any
client is obtained, and verbose response checking.
### API and Format
`FileIO` gains one default method that returns empty and touches no storage. `BatchFileDeleter` and
`BatchDeleteResult` are new public types. No file format change.
### Documentation
None. This adds no option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][oss] Add strict batch deletion to FileIO [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #9398:
URL: https://github.com/apache/paimon/pull/9398#issuecomment-5429133225
Closing this. The runtime on the path that motivated it does not delete in batches, so the OSS provider here would never be reached there, and the bounded cleanup concurrency in #9397 already covers that case.
Its only consumer, #9424, is closed with it. The branch stays if this becomes worth picking up again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][oss] Add strict batch deletion to FileIO [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng closed pull request #9398: [common][oss] Add strict batch deletion to FileIO
URL: https://github.com/apache/paimon/pull/9398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org