You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "swuferhong (via GitHub)" <gi...@apache.org> on 2026/07/09 03:58:39 UTC

[I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

swuferhong opened a new issue, #3625:
URL: https://github.com/apache/fluss/issues/3625

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Fluss version
   
   0.9.0 (latest release)
   
   ### Please describe the bug 🐞
   
   ## Description
   
   Coordinator-side dynamic configs may become stale on an already-running standby `CoordinatorServer`. The coordinator `DynamicConfigManager` currently fetches dynamic configs from ZooKeeper during startup, but coordinator instances ignore later config change notifications. As a result, a standby coordinator does not continuously track the latest dynamic configs while it remains standby.
   
   When that standby later becomes the active coordinator, leader-only components are initialized and registered against the local `DynamicConfigManager`. If the local `DynamicServerConfig.currentConfig` is stale, those components may be configured with old values even though ZooKeeper already contains newer dynamic configs.
   
   This is not specific to one config option. It can affect coordinator-side dynamic configs in general. The issue is easier to observe with `coordinator.rebalance.max-inflight-tasks`: if users dynamically set it to `0` to pause rebalance scheduling, a coordinator failover may cause the new active coordinator to initialize `RebalanceManager` with the old value, such as the default `1`, and resume scheduling rebalance tasks unexpectedly.
   
   ## Impact
   
   After coordinator failover, the new active coordinator may not respect the latest dynamic configs stored in ZooKeeper.
   
   ## Root Cause
   
   Coordinator `DynamicConfigManager` uses `isCoordinator = true`, and the config change notification handler returns early for coordinator instances. Therefore, standby coordinators do not apply dynamic config changes after their initial startup fetch.
   
   Leader-only components, such as `RebalanceManager`, are registered when the coordinator becomes leader. Their initial configuration comes from the local `DynamicServerConfig.currentConfig`, which may be stale on a long-running standby.
   
   ## Proposed Solution
   
   Refresh dynamic configs from ZooKeeper when a standby CoordinatorServer becomes leader, before registering leader-only components.
   
   Specifically:
   
   1. Add a method to `DynamicConfigManager`, for example:
   
      ```java
      public void refreshFromZooKeeper() throws Exception {
          Map<String, String> entityConfigs = zooKeeperClient.fetchEntityConfig();
          dynamicServerConfig.updateDynamicConfig(entityConfigs, true);
      }
   
   ### Solution
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "fxbing (via GitHub)" <gi...@apache.org>.
fxbing commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4927086211

   @swuferhong If @fhan688  hasn’t started on this yet, I’d be happy to take it.
   
   I’ll follow the proposed direction: refresh coordinator dynamic configs from ZooKeeper during standby-to-leader transition and add a regression test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4957805096

   > Hi, [@fhan688](https://github.com/fhan688) do you want to take it?
   
   Sorry for reply late. I see there's already been a submission. Overall, I prefer @loserwang1024 's later proposal and I suggest we confirm the proposal first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4921469701

   Hi, @fhan688 do you want to take it? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4957894953

   > > Hi, [@fhan688](https://github.com/fhan688) do you want to take it?
   > 
   > Sorry for reply late. I see there's already been a submission. Overall, I prefer [@loserwang1024](https://github.com/loserwang1024) 's later proposal and I suggest we confirm the proposal first. [@swuferhong](https://github.com/swuferhong)
   
   By the way, I'm currently implementing based on Solution 2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 closed issue #3625: [server] Coordinator standby may use stale dynamic configs after failover
URL: https://github.com/apache/fluss/issues/3625


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "fxbing (via GitHub)" <gi...@apache.org>.
fxbing commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4958452489

   > Hi, [@fxbing](https://github.com/fxbing) Thanks a lot for your contribution! As this issue was reported by [@fhan688](https://github.com/fhan688) offline, and he started working on a fix right when the issue was created, so I will assign it to him. I'll ping you when there are new issues come up, really thanks for your contribution.
   
   @swuferhong No worries, thanks for letting me know! Happy to help with other issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4958365953

   > > Hi, [@fhan688](https://github.com/fhan688) do you want to take it?
   > 
   > Sorry for reply late. I see there's already been a submission. Overall, I prefer [@loserwang1024](https://github.com/loserwang1024) 's later proposal and I suggest we confirm the proposal first. [@swuferhong](https://github.com/swuferhong)
   
   I agree with @fhan688 than we need to confirm the proposal first. In my opinion,  +1 for the option: the HA standby also synchronously applies the update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4955324668

   There are two approaches: refreshing the dynamic configuration when becoming a leader, or having followers also accept configuration change notifications. I prefer the later, because certain connection-related configuration items (such as adding new users) should take effect immediately. It's undesirable for different nodes to use different usernames and passwords.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [server] Coordinator standby may use stale dynamic configs after failover [fluss]

Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on issue #3625:
URL: https://github.com/apache/fluss/issues/3625#issuecomment-4958408764

   Hi, @fxbing Thanks a lot for your contribution! As this issue was reported by @fhan688 offline, and he started working on a fix right when the issue was created, so I will assign it to him. I'll ping you when there are new issues come up, really thanks for your contribution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org