You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dragosvictor (via GitHub)" <gi...@apache.org> on 2023/12/04 20:43:06 UTC
Re: [PR] [improve][broker] PIP-307 Expose tombstone and inflight state waiting time configs. Handle AddEntry failure during topic transfer [pulsar]
dragosvictor commented on code in PR #21668:
URL: https://github.com/apache/pulsar/pull/21668#discussion_r1414453875
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -200,33 +197,25 @@ enum MetadataState {
Unstable
}
- public static ServiceUnitStateChannelImpl newInstance(PulsarService pulsar) {
- return new ServiceUnitStateChannelImpl(pulsar);
- }
-
- public ServiceUnitStateChannelImpl(PulsarService pulsar) {
- this(pulsar, MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS, OWNERSHIP_MONITOR_DELAY_TIME_IN_SECS);
- }
-
@VisibleForTesting
- public ServiceUnitStateChannelImpl(PulsarService pulsar,
- long inFlightStateWaitingTimeInMillis,
- long ownershipMonitorDelayTimeInSecs) {
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
this.pulsar = pulsar;
this.config = pulsar.getConfig();
this.lookupServiceAddress = pulsar.getLookupServiceAddress();
this.schema = Schema.JSON(ServiceUnitStateData.class);
this.getOwnerRequests = new ConcurrentHashMap<>();
this.cleanupJobs = new ConcurrentHashMap<>();
this.stateChangeListeners = new StateChangeListeners();
- this.semiTerminalStateWaitingTimeInMillis = config.getLoadBalancerServiceUnitStateTombstoneDelayTimeInSeconds()
+ this.stateTombstoneDelayTimeInSeconds = config.getLoadBalancerServiceUnitStateTombstoneDelayTimeInSeconds()
* 1000;
Review Comment:
Nit: this shouldn't be multiplied by 1000 anymore.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java:
##########
@@ -468,8 +459,11 @@ public void testTransferClientReconnectionWithoutLookup(boolean isPersistentTopi
throw new RuntimeException(e);
}
});
- assertTrue(producer.isConnected());
- verify(lookup, times(lookupCountBeforeUnload)).getBroker(topicName);
+
+ Awaitility.await().atMost(5, TimeUnit.SECONDS).untilAsserted(() -> {
+ assertTrue(producer.isConnected());
+ verify(lookup, times(lookupCountBeforeUnload)).getBroker(topicName);
+ });
Review Comment:
Can we simplify as suggested below? We aren't expecting any new calls to `lookup.getBroker(topicName)`, we could go further and replace the condition with `never()`.
```suggestion
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(producer::isConnected);
verify(lookup, times(lookupCountBeforeUnload)).getBroker(topicName);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##########
@@ -652,6 +652,20 @@ public void addComplete(Position pos, ByteBuf entryData, Object ctx) {
@Override
public synchronized void addFailed(ManagedLedgerException exception, Object ctx) {
PublishContext callback = (PublishContext) ctx;
+
+ /* If the topic is being transferred(in the Releasing bundle state),
+ we don't want to forcefully close topic here.
+ Instead, we will rely on the service unit state channel's bundle(topic) transfer protocol.
+ At the end of the transfer protocol, at Owned state, the source broker should close the topic properly.
+ */
+ if (isClosingOrDeleting
+ && ExtensibleLoadManagerImpl.isLoadManagerExtensionEnabled(getBrokerService().pulsar())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to persist msg in store: {} while closing or deleting.",
+ topic, exception.getMessage(), exception);
+ }
+ return;
+ }
Review Comment:
Looks correct, but is there anything to be done about the `PublishContext` callback?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -200,33 +197,25 @@ enum MetadataState {
Unstable
}
- public static ServiceUnitStateChannelImpl newInstance(PulsarService pulsar) {
- return new ServiceUnitStateChannelImpl(pulsar);
- }
-
- public ServiceUnitStateChannelImpl(PulsarService pulsar) {
- this(pulsar, MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS, OWNERSHIP_MONITOR_DELAY_TIME_IN_SECS);
- }
-
@VisibleForTesting
- public ServiceUnitStateChannelImpl(PulsarService pulsar,
- long inFlightStateWaitingTimeInMillis,
- long ownershipMonitorDelayTimeInSecs) {
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
this.pulsar = pulsar;
this.config = pulsar.getConfig();
this.lookupServiceAddress = pulsar.getLookupServiceAddress();
this.schema = Schema.JSON(ServiceUnitStateData.class);
this.getOwnerRequests = new ConcurrentHashMap<>();
this.cleanupJobs = new ConcurrentHashMap<>();
this.stateChangeListeners = new StateChangeListeners();
- this.semiTerminalStateWaitingTimeInMillis = config.getLoadBalancerServiceUnitStateTombstoneDelayTimeInSeconds()
+ this.stateTombstoneDelayTimeInSeconds = config.getLoadBalancerServiceUnitStateTombstoneDelayTimeInSeconds()
* 1000;
- this.inFlightStateWaitingTimeInMillis = inFlightStateWaitingTimeInMillis;
- this.ownershipMonitorDelayTimeInSecs = ownershipMonitorDelayTimeInSecs;
- if (semiTerminalStateWaitingTimeInMillis < inFlightStateWaitingTimeInMillis) {
+ this.inFlightStateWaitingTimeInMillis = config.getLoadBalancerInFlightServiceUnitStateWaitingTimeInMillis();
+ this.ownershipMonitorDelayTimeInSecs = config.getLoadBalancerServiceUnitStateMonitorIntervalInSeconds();
+ if (stateTombstoneDelayTimeInSeconds < inFlightStateWaitingTimeInMillis) {
throw new IllegalArgumentException(
Review Comment:
Nit: comparing seconds to milliseconds.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -1457,7 +1446,7 @@ protected void monitorOwnerships(List<String> brokers) {
continue;
}
- if (now - stateData.timestamp() > semiTerminalStateWaitingTimeInMillis) {
+ if (now - stateData.timestamp() > stateTombstoneDelayTimeInSeconds) {
Review Comment:
Nit: comparing seconds to milliseconds.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -654,12 +654,12 @@ public void splitAndRetryTest() throws Exception {
FieldUtils.writeDeclaredField(channel1,
"inFlightStateWaitingTimeInMillis", 30 * 1000, true);
FieldUtils.writeDeclaredField(channel1,
- "semiTerminalStateWaitingTimeInMillis", 300 * 1000, true);
+ "stateTombstoneDelayTimeInSeconds", 300 * 1000, true);
Review Comment:
Here and below: are these values still correct, since we switched to a different time unit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org