[I] [Bug] Coordinator metrics can stop reporting after failed partition cleanup [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg opened a new issue, #4088:
URL: https://github.com/apache/fluss/issues/4088

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Fluss version
   
   0.9.0 (latest release)
   
   ### Please describe the bug 🐞
   
   When a primary-key table partition is dropped, `CoordinatorEventProcessor.processDropPartition` removes its `CompletedSnapshotStore` instances before unregistering the corresponding metric group.
   
   If a later cleanup step throws, the drop event terminates with the snapshot stores already removed while the gauges remain registered. Subsequent Prometheus scrapes invoke these residual gauges, which look up their backing stores through `bucketCompletedSnapshotStores`. Because the stores no longer exist, the gauge callback throws a `NullPointerException`.
   
   Every subsequent scrape encounters the same residual state, so Coordinator metrics can remain unavailable until the process is restarted.
   
   There is also an in-flight scrape race. A reporter may already hold a gauge reference when metric-group unregistration begins. Unregistration alone does not make that callback safe because the gauge resolves its store through the mutable manager map.
   
   ### Reproduction
   
   1. Create a partitioned primary-key table and its snapshot-store gauges.
   2. Delete the partition in ZooKeeper, allowing the watcher, event throttler, and Coordinator event queue to invoke `processDropPartition`.
   3. Inject a failure in a cleanup step after snapshot-store removal but before metric-group removal.
   4. Wait for the event to finish.
   5. Scrape both the active registry gauges and gauge references captured by an in-flight scrape.
   
   
   ### Solution
   
   Unregister the table or partition metric group before removing its snapshot stores.
   
   Additionally, bind each snapshot gauge directly to the `CompletedSnapshotStore` created for that bucket instead of resolving it through the manager map on every scrape. This removes the residual registered state and keeps callbacks captured by an in-flight scrape valid during concurrent unregistration.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Coordinator metrics can stop reporting after failed partition cleanup [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg closed issue #4088: [Bug] Coordinator metrics can stop reporting after failed partition cleanup
URL: https://github.com/apache/fluss/issues/4088


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org