You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "zymap (via GitHub)" <gi...@apache.org> on 2026/03/12 06:46:35 UTC
[PR] Count the connection failure as the condition of quarantine [bookkeeper]
zymap opened a new pull request, #4727:
URL: https://github.com/apache/bookkeeper/pull/4727
---
### Motivation
Currently, the bookie client quarantine mechanism primarily triggers based on `read` and `write` error responses from Bookies. However, in multi-region deployments, a common failure mode is the **Network Partition** or **DNS Resolution Failure** at the Region level.
In such scenarios:
1. A Bookie remains registered in ZooKeeper (it can still heartbeat to its local ZK observer).
2. The Client (Broker) cannot resolve the Bookie's IP or establish a TCP connection.
3. The `EnsemblePlacementPolicy` (especially `RegionAwareEnsemblePlacementPolicy`) sees the Bookie as "Available" and repeatedly selects it to satisfy `minRack` or `E/Qw` constraints.
4. The `LedgerHandle` fails to write because it cannot initialize a connection handle, triggering an **Ensemble Change**.
5. Because the connection failure didn't trigger a quarantine, the placement policy picks the **same problematic Bookie** again in the next iteration.
This creates an **infinite Ensemble Change loop**, causing the Ledger write to hang indefinitely and bloating the Ledger metadata in ZooKeeper with thousands of segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Count the connection failure as the condition of quarantine [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4727:
URL: https://github.com/apache/bookkeeper/pull/4727#discussion_r2958459136
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:
##########
@@ -1456,6 +1457,28 @@ public ClientConfiguration setBookieErrorThresholdPerInterval(long thresholdPerI
return this;
}
+
+ /**
+ * Set if count the bookie connecting error into the quarantine condition. If this is enabled, the connection
+ * error will be counted into the BookieErrorThresholdPerInterval. So be careful to set the quarantine time.
+ *
+ * @param enabled
+ * @return
+ */
+ public ClientConfiguration setBookieConnectionErrorQuarantineEnabled(boolean enabled) {
+ setProperty(BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED, enabled);
Review Comment:
```suggestion
setProperty(BOOKIE_CONNECTION_ERROR_QUARANTINE_ENABLED, enabled);
```
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:
##########
@@ -1456,6 +1457,28 @@ public ClientConfiguration setBookieErrorThresholdPerInterval(long thresholdPerI
return this;
}
+
+ /**
+ * Set if count the bookie connecting error into the quarantine condition. If this is enabled, the connection
+ * error will be counted into the BookieErrorThresholdPerInterval. So be careful to set the quarantine time.
+ *
+ * @param enabled
+ * @return
+ */
+ public ClientConfiguration setBookieConnectionErrorQuarantineEnabled(boolean enabled) {
+ setProperty(BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED, enabled);
+ return this;
+ }
+
+ /**
+ * Get if count the bookie connecting error into the quarantine condition.
+ *
+ * @return
+ */
+ public boolean getBookieConnectionErrorQuarantineEnabled() {
+ return getBoolean(BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED, false);
Review Comment:
```suggestion
return getBoolean(BOOKIE_CONNECTION_ERROR_QUARANTINE_ENABLED, false);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Count the connection failure as the condition of quarantine [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4727:
URL: https://github.com/apache/bookkeeper/pull/4727#discussion_r2958457240
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java:
##########
@@ -1818,6 +1818,9 @@ public void operationComplete(ChannelFuture future) {
if (state != ConnectionState.CLOSED) {
state = ConnectionState.DISCONNECTED;
}
+ if (conf.getBookieConnectingErrorCountedIntoQuarantine()) {
Review Comment:
```suggestion
if (conf.getBookieConnectionErrorQuarantineEnabled()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Count the connection failure as the condition of quarantine [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4727:
URL: https://github.com/apache/bookkeeper/pull/4727#discussion_r2929533507
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:
##########
@@ -1456,6 +1457,28 @@ public ClientConfiguration setBookieErrorThresholdPerInterval(long thresholdPerI
return this;
}
+
+ /**
+ * Set if count the bookie connecting error into the quarantine condition. If this is enabled, the connection
+ * error will be counted into the BookieErrorThresholdPerInterval. So be careful to set the quarantine time.
+ *
+ * @param enabled
+ * @return
+ */
+ public ClientConfiguration setBookieConnectingErrorCountedIntoQuarantine(boolean enabled) {
+ setProperty(BOOKIE_CONNECTING_ERROR_COUNTED_INTO_QUARANTINE, enabled);
+ return this;
+ }
+
+ /**
+ * Get if count the bookie connecting error into the quarantine condition.
+ *
+ * @return
+ */
+ public boolean getBookieConnectingErrorCountedIntoQuarantine() {
+ return getBoolean(BOOKIE_CONNECTING_ERROR_COUNTED_INTO_QUARANTINE);
Review Comment:
Default value should be provided as the 2nd argument, for example `getBoolean(BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED, false)` (example with the suggested name of the config)
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:
##########
@@ -140,6 +140,7 @@ public class ClientConfiguration extends AbstractConfiguration<ClientConfigurati
protected static final String BOOKIE_ERROR_THRESHOLD_PER_INTERVAL = "bookieErrorThresholdPerInterval";
protected static final String BOOKIE_QUARANTINE_TIME_SECONDS = "bookieQuarantineTimeSeconds";
protected static final String BOOKIE_QUARANTINE_RATIO = "bookieQuarantineRatio";
+ protected static final String BOOKIE_CONNECTING_ERROR_COUNTED_INTO_QUARANTINE = "bookieConnectingErrorCountedIntoQuarantine";
Review Comment:
Could we call this `bookieConnectionErrorQuarantineEnabled` instead? The "counted into" part is slightly confusing.
```suggestion
protected static final String BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED = "bookieConnectionErrorQuarantineEnabled";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Count the connection failure as the condition of quarantine [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4727:
URL: https://github.com/apache/bookkeeper/pull/4727#discussion_r2958458175
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:
##########
@@ -140,6 +140,7 @@ public class ClientConfiguration extends AbstractConfiguration<ClientConfigurati
protected static final String BOOKIE_ERROR_THRESHOLD_PER_INTERVAL = "bookieErrorThresholdPerInterval";
protected static final String BOOKIE_QUARANTINE_TIME_SECONDS = "bookieQuarantineTimeSeconds";
protected static final String BOOKIE_QUARANTINE_RATIO = "bookieQuarantineRatio";
+ protected static final String BOOKIE_CONNECTING_ERROR_QUARANTINE_ENABLED = "bookieConnectionErrorQuarantineEnabled";
Review Comment:
```suggestion
protected static final String BOOKIE_CONNECTION_ERROR_QUARANTINE_ENABLED = "bookieConnectionErrorQuarantineEnabled";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Count the connection failure as the condition of quarantine [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #4727:
URL: https://github.com/apache/bookkeeper/pull/4727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org