You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "wuchong (via GitHub)" <gi...@apache.org> on 2026/01/25 09:18:44 UTC
[I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
wuchong opened a new issue, #2473:
URL: https://github.com/apache/fluss/issues/2473
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
https://github.com/apache/fluss/actions/runs/21329609454/job/61392275442
```
Error: Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.961 s <<< FAILURE! - in org.apache.fluss.client.security.acl.FlussAuthorizationITCase
Error: org.apache.fluss.client.security.acl.FlussAuthorizationITCase.testRebalance Time elapsed: 4.054 s <<< ERROR!
java.util.concurrent.ExecutionException: org.apache.fluss.exception.RebalanceFailureException: Failed to generate plan and execute rebalance. The root cause: Leader replica is not in the bucket.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2005)
at org.apache.fluss.client.security.acl.FlussAuthorizationITCase.testRebalance(FlussAuthorizationITCase.java:964)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInte
```
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #2473:
URL: https://github.com/apache/fluss/issues/2473#issuecomment-3796257129
cc @swuferhong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on issue #2473:
URL: https://github.com/apache/fluss/issues/2473#issuecomment-3796778413
Hi, @ZuebeyirEser. I haven't looked into this issue yet; you can reproduce it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
Posted by "ZuebeyirEser (via GitHub)" <gi...@apache.org>.
ZuebeyirEser commented on issue #2473:
URL: https://github.com/apache/fluss/issues/2473#issuecomment-3796867119
@swuferhong I’ve been looking into this and was able to reproduce the error locally.
It appears to be a transient metadata state. When `rebalance()` is called shortly after a table is created, the `CoordinatorContext` sometimes has the leader elected, but the replica assignment list hasn't been fully updated yet. This causes the optimizer to hit the Leader replica is not in the bucket check.
I wrote a small loop to trigger rebalance during concurrent table creation. Without the change, it failed consistently within a few iterations. With the fix, it successfully completed 50 iterations.
```java
@Test
void testRebalanceDuringConcurrentTableCreation() throws Exception {
rootAdmin.createAcls(Collections.singletonList(
new AclBinding(
Resource.cluster(),
new AccessControlEntry(
guestPrincipal,
WILD_CARD_HOST,
OperationType.WRITE,
PermissionType.ALLOW))))
.all().get();
for (int i = 0; i < 20; i++) {
TablePath transientTable = TablePath.of("test_db_1", "transient_rebalance_table_" + i);
rootAdmin.createTable(transientTable, DATA1_TABLE_DESCRIPTOR_PK, false);
// Verification: ensure rebalance doesn't fail due to transient metadata
assertThatCode(() -> guestAdmin.rebalance(Collections.emptyList()).get())
.doesNotThrowAnyException();
rootAdmin.dropTable(transientTable, true).get();
}
}
```
My proposed solution:
I’ve tested a change in RebalanceManager to gracefully skip buckets that are in this transient state (where the leader is -1 or not yet in the assignment list). This allows the rebalance to continue for all other stable tables. As far as I can see, this is safe because the skipped bucket will simply be picked up in the next rebalance cycle once its state is consistent.
I'd be happy to submit a PR with this fix and the regression test if this approach sounds reasonable to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
Posted by "ZuebeyirEser (via GitHub)" <gi...@apache.org>.
ZuebeyirEser commented on issue #2473:
URL: https://github.com/apache/fluss/issues/2473#issuecomment-3796442450
Hi @wuchong ,
Since I've been working on test stability regarding replica/ISR timing (in the AdjustIsrITCase PR), I'd be interested in investigating this one as well.
It looks like another potential race condition during the rebalance operation. If @swuferhong isn't already looking into it, I'm happy to pick this up and try to reproduce/fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unstable test FlussAuthorizationITCase.testRebalance [fluss]
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong closed issue #2473: Unstable test FlussAuthorizationITCase.testRebalance
URL: https://github.com/apache/fluss/issues/2473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org