You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "wuchong (via GitHub)" <gi...@apache.org> on 2026/09/10 12:33:42 UTC
[I] [client] Keep dynamic partition writes non-blocking with asynchronous metadata refresh [fluss]
wuchong opened a new issue, #4287:
URL: https://github.com/apache/fluss/issues/4287
### Search before asking
- [x] I searched the existing issues. The related metadata work in #2209 is linked below; this issue tracks the dynamic-partition write-path follow-up.
### Description
Follow up on [this review discussion in PR #3908](https://github.com/apache/fluss/pull/3908#discussion_r3969478604).
**This work is explicitly deferred to a separate follow-up PR after #3908. It is not required to be implemented within #3908.**
Per-partition bucket counts require a writer to distinguish a partition's actual layout from the table-level default. The reviewed implementation waits for the partition ID and bucket count in `DynamicPartitionCreator.waitForPartitionMetadata()`, including synchronous metadata refreshes and backoff. Doing this on the application write thread can turn dynamic partition creation or delayed metadata availability into a write-latency regression.
The discussion proposes keeping partition creation asynchronous and checking the actual bucket count at send time, similarly to resolving bucket locations. A batch whose expected routing count no longer matches can fail explicitly. Bucket-count changes are infrequent, so rejecting affected batches during that transition is preferable to making writes wait for metadata. Asynchronous client metadata refresh can further reduce this window by making the actual partition bucket count available before subsequent batches are built.
### Proposed follow-up
1. Keep dynamic partition creation and routing-metadata acquisition off the application write thread. Missing metadata should trigger asynchronous work rather than a synchronous wait for partition readiness.
2. Refresh and publish partition identity and actual bucket-count metadata asynchronously. Coalesce concurrent refresh requests for the same partition and let new batches use the refreshed layout once available.
3. Preserve the bucket count used when a batch was routed. At send time, compare it with the actual partition count. If they differ, complete the affected batch exceptionally with a clear routing-mismatch error; do not silently substitute a new count while keeping the old bucket ID.
4. Invalidate stale routing/assignment state so subsequent batches can be built with current metadata. One implementation direction discussed in the review is to report invalid paths through an `invalidBucketRoutingTables`-style result from `RecordAccumulator` and let `Sender` fail the affected batches. The exact representation should follow the existing ownership and synchronization model.
5. Preserve timeout, partition-creation failure, and writer-close behavior, and keep writes to unrelated ready partitions progressing while another partition's metadata is being refreshed.
Relevant components: `DynamicPartitionCreator`, `MetadataUpdater`, `RecordAccumulator`, and `Sender`.
### Validation
- With metadata responses or partition creation deliberately delayed, `append`/`upsert` must not wait on the application thread for routing metadata, assuming write-buffer space is available. Verify that writes to an already-ready partition can still progress.
- Concurrent writes to the same not-yet-ready partition share creation/refresh work rather than issuing one refresh per record.
- After ALTER changes the table default, old and new partitions retain their own counts. New batches use the correct partition layout once the asynchronous refresh completes.
- A batch formed with a stale count is rejected by `Sender`; unrelated valid batches remain successful, and no records are silently sent under a different routing layout.
- Cover creation failure, refresh failure/timeout, and close during an in-flight refresh, with deterministic completion and no stranded write futures.
### Related work
- [PR #3908 discussion](https://github.com/apache/fluss/pull/3908#discussion_r3969478604): motivation and the proposed send-time mismatch handling.
- [Issue #2209](https://github.com/apache/fluss/issues/2209): asynchronous metadata updates, batching, and deduplication to prevent lookup-side Netty IO-thread blocking. Reuse or coordinate that shared infrastructure; this follow-up specifically covers dynamic partition writes and bucket-routing readiness.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [client] Keep dynamic partition writes non-blocking with asynchronous metadata refresh [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong closed issue #4287: [client] Keep dynamic partition writes non-blocking with asynchronous metadata refresh
URL: https://github.com/apache/fluss/issues/4287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [client] Keep dynamic partition writes non-blocking with asynchronous metadata refresh [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #4287:
URL: https://github.com/apache/fluss/issues/4287#issuecomment-5653976331
https://github.com/apache/fluss/pull/4309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org