You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "BewareMyPower (via GitHub)" <gi...@apache.org> on 2024/02/27 10:11:21 UTC
[PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
BewareMyPower opened a new pull request, #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406
Fixes https://github.com/apache/pulsar-client-cpp/issues/405
### Motivation
After triggering a blue-green migration, the socket will be disconnected and then schedule a reconnection to the blue cluster. However, the blue cluster could never respond with a response for Producer or Subscribe commands. Take producer as example, it means `connectionOpened` will not complete and `reconnectionPending_` will not become false.
Then, after receiving a `CommandProducerClose` command from the blue cluster, a new reconnection will be scheduled to the green cluster but it will be skipped because `reconnectionPending_` is true, which means the previous `connectionOpened` future is not completed until the 30s timeout is reached.
```
2024-02-26 06:09:30.251 INFO [139737465607744] HandlerBase:101 | [persistent://public/unload-test/topic-1708927732, sub, 0] Ignoring reconnection attempt since there's already a pending reconnection
2024-02-26 06:10:00.035 WARN [139737859880512] ProducerImpl:291 | [persistent://public/unload-test/topic-1708927732, cluster-a-0-0] Failed to reconnect producer: TimeOut
```
### Modifications
When receiving the `TOPIC_MIGRATED` command, cancel the pending `Producer` and `Subscribe` commands so that `connectionOpened` will fail with a retryable error. In the next time of reconnection, the green cluster will be connected.
Fix the `ExtensibleLoadManagerTest` with a more strict timeout check. After this change, it will pass in about 3 seconds locally, while in CI even if it passed, it takes about 70 seconds before.
Besides, fix the possible crash on macOS when closing the client, see https://github.com/apache/pulsar-client-cpp/issues/405#issuecomment-1963969215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406#discussion_r1505232798
##########
lib/ProducerImpl.h:
##########
@@ -213,6 +215,7 @@ class ProducerImpl : public HandlerBase, public ProducerImplBase {
ProducerInterceptorsPtr interceptors_;
bool retryOnCreationError_;
+ std::atomic<long> producerRequestId_{0L};
Review Comment:
Oh Yes, it's right. I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406#issuecomment-1966556945
The test is still flaky when duplicated messages are received.
Logs in CI:
```
2024-02-27 10:25:03.553 INFO [139979829278272] ExtensibleLoadManagerTest:136 | receive i: 2 10 ms
2024-02-27 10:25:03.554 INFO [139979829278272] ExtensibleLoadManagerTest:143 | acked i:2 0 ms
2024-02-27 10:25:03.559 INFO [139980456872960] ExtensibleLoadManagerTest:183 | before lookup responseData:{"brokerUrl":"pulsar://broker-2:6650","httpUrl":"http://broker-2:8080","nativeUrl":"pulsar://broker-2:6650","brokerUrlSsl":""},unload url:http://localhost:8080/admin/v2/namespaces/public/unload-test/0x00000000_0xffffffff/unload?destinationBroker=broker-1:8080,lookupCountBeforeUnload:2
2024-02-27 10:25:08.555 INFO [139979829278272] ExtensibleLoadManagerTest:136 | receive i: 2 5000 ms
```
Logs locally:
```
2024-02-27 21:25:29.831 INFO [0x170017000] ExtensibleLoadManagerTest:115 | produce i: 7 7 ms
2024-02-27 21:25:29.833 INFO [0x1e9aa2100] ExtensibleLoadManagerTest:184 | before lookup responseData:{"brokerUrl":"pulsar://broker-2:6650","httpUrl":"http://broker-2:8080","nativeUrl":"pulsar://broker-2:6650","brokerUrlSsl":""},unload url:http://localhost:8080/admin/v2/namespaces/public/unload-test/0x00000000_0xffffffff/unload?destinationBroker=broker-1:8080,lookupCountBeforeUnload:2
2024-02-27 21:25:29.833 INFO [0x1700a3000] ExtensibleLoadManagerTest:136 | receive i: 6 10 ms
2024-02-27 21:25:29.833 INFO [0x1700a3000] ExtensibleLoadManagerTest:143 | acked i:6 0 ms
2024-02-27 21:25:29.833 INFO [0x1700a3000] ExtensibleLoadManagerTest:136 | receive i: 7 0 ms
2024-02-27 21:25:29.833 INFO [0x1700a3000] ExtensibleLoadManagerTest:143 | acked i:7 0 ms
2024-02-27 21:25:34.833 INFO [0x1700a3000] ExtensibleLoadManagerTest:136 | receive i: 7 5000 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower merged PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406#discussion_r1504571584
##########
lib/ProducerImpl.h:
##########
@@ -213,6 +215,7 @@ class ProducerImpl : public HandlerBase, public ProducerImplBase {
ProducerInterceptorsPtr interceptors_;
bool retryOnCreationError_;
+ std::atomic<long> producerRequestId_{0L};
Review Comment:
Should we init to something else like -1?
##########
lib/ConsumerImpl.h:
##########
@@ -310,6 +312,7 @@ class ConsumerImpl : public ConsumerImplBase {
std::atomic_bool expireChunkMessageTaskScheduled_{false};
ConsumerInterceptorsPtr interceptors_;
+ std::atomic<long> subscribeRequestId_;
Review Comment:
Should we init to something else like -1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406#issuecomment-1966714199
The flakiness in the current test is related to the broker side (https://github.com/apache/pulsar/issues/22136) so I mark this PR as ready to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix blue-green migration might be stuck due to an existing reconnection [pulsar-client-cpp]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #406:
URL: https://github.com/apache/pulsar-client-cpp/pull/406#issuecomment-1968663538
@heesung-sn I moved the field into `HandlerBase` as `firstRequestIdAfterConnect_` and initialized it with -1. PTAL again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org