You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/18 11:19:36 UTC
[GitHub] [pulsar] BewareMyPower opened a new pull request, #17162: [feat][cpp] Support multiple brokers in service URL
BewareMyPower opened a new pull request, #17162:
URL: https://github.com/apache/pulsar/pull/17162
### Motivation
It's a catchup for https://github.com/apache/pulsar/pull/3249.
Currently C++ client doesn't have strict validation on the service URL.
If multiple brokers are specified, only the 1st broker will be used.
### Modifications
- Add `ServiceURI` to support configuring multiple brokers in the
service URL. See `ServiceURITest` for how to configure it.
- Add `ServiceNameResolver` whose `resolveHost` method selects the
next broker in a round robin way.
- Since the broker's address for topic lookup is no longer the service
URL itself, to handle the case when proxy is enabled, a `getBroker`
method (like the same method in Java's `LookupService`), which returns
the future of a pair of logical and physical addresses, is added to
replace `lookupAsync`.
- Apply `ServiceNameResolver` into `ClientImpl` and the
`LookupService` implementations.
- Rename `BinaryLookupServiceTest` to `LookupServiceTest` and add
`testMultiAddresses` to test both `BinaryProtoLookupService` and
`HTTPLookupService` that all available brokers will be accessed in a
round robin way.
### Behavior Change
This PR also changes the behavior when the user passed service URL.
Before this PR, creating a `Client` object will succeed but the methods to
create a producer or consumer will return a failed `Result`. After this PR,
an `invalid_argument` exception will be thrown.
### TODO
This is the 1st part to support multiple brokers. Even with this patch,
if one of these hosts is not available, the topic lookup will fail
immediately instead of trying other broker. Since this patch already
includes many code changes, I will push another patch to fix it.
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
Fixes #<xyz>
*(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>` to link to the master issue.)*
Master Issue: #<xyz>
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve.*
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#issuecomment-1220253037
Draft this PR until the Python tests are fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower merged pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower merged PR #17162:
URL: https://github.com/apache/pulsar/pull/17162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#discussion_r952743129
##########
pulsar-client-cpp/tests/ServiceURITest.cc:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <gtest/gtest.h>
+#include "lib/ServiceURI.h"
+
+using namespace pulsar;
+
+static void verifyServiceURIFailure(const std::string& uriString, const std::string& errorMsg) {
+ try {
+ ServiceURI uri{uriString};
+ std::cerr << uriString << " should be invalid" << std::endl;
+ FAIL();
+ } catch (const std::invalid_argument& e) {
+ EXPECT_EQ(errorMsg, e.what());
+ }
+}
+
+static void verifyServiceURI(const std::string& uriString, PulsarScheme expectedScheme,
+ const std::vector<std::string>& expectedServiceHosts) {
+ ServiceURI uri{uriString};
+ EXPECT_EQ(uri.getScheme(), expectedScheme);
+ EXPECT_EQ(uri.getServiceHosts(), expectedServiceHosts);
+}
+
+TEST(ServiceURITest, testInvalidServiceUris) {
+ verifyServiceURIFailure("localhost:6650", "The scheme part is missing: localhost:6650");
Review Comment:
Look like we don't support IPv6 yet. Do we have a plan to support it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#discussion_r953265166
##########
pulsar-client-cpp/tests/ServiceURITest.cc:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <gtest/gtest.h>
+#include "lib/ServiceURI.h"
+
+using namespace pulsar;
+
+static void verifyServiceURIFailure(const std::string& uriString, const std::string& errorMsg) {
+ try {
+ ServiceURI uri{uriString};
+ std::cerr << uriString << " should be invalid" << std::endl;
+ FAIL();
+ } catch (const std::invalid_argument& e) {
+ EXPECT_EQ(errorMsg, e.what());
+ }
+}
+
+static void verifyServiceURI(const std::string& uriString, PulsarScheme expectedScheme,
+ const std::vector<std::string>& expectedServiceHosts) {
+ ServiceURI uri{uriString};
+ EXPECT_EQ(uri.getScheme(), expectedScheme);
+ EXPECT_EQ(uri.getServiceHosts(), expectedServiceHosts);
+}
+
+TEST(ServiceURITest, testInvalidServiceUris) {
+ verifyServiceURIFailure("localhost:6650", "The scheme part is missing: localhost:6650");
Review Comment:
Not yet. We should make multiple brokers feature available first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#issuecomment-1219523957
The `testWrongListener` is affected, I will mark it as draft and fix it.
```
2022-08-18 13:23:38.989 INFO [140182675973888] ConnectionPool:96 | Created connection for
/pulsar/pulsar-client-cpp/tests/ClientTest.cc:248: Failure
Expected equality of these values:
ResultServiceUnitNotReady
Which is: ServiceUnitNotReady
client.createProducer(topic, producer)
Which is: ConnectError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#discussion_r958035181
##########
pulsar-client-cpp/lib/ClientImpl.cc:
##########
@@ -414,36 +402,32 @@ void ClientImpl::handleConsumerCreated(Result result, ConsumerImplBaseWeakPtr co
Future<Result, ClientConnectionWeakPtr> ClientImpl::getConnection(const std::string& topic) {
Promise<Result, ClientConnectionWeakPtr> promise;
- lookupServicePtr_->lookupAsync(topic).addListener(std::bind(&ClientImpl::handleLookup, shared_from_this(),
- std::placeholders::_1, std::placeholders::_2,
- promise));
- return promise.getFuture();
-}
-void ClientImpl::handleLookup(Result result, LookupDataResultPtr data,
- Promise<Result, ClientConnectionWeakPtr> promise) {
- if (data) {
- const std::string& logicalAddress =
- clientConfiguration_.isUseTls() ? data->getBrokerUrlTls() : data->getBrokerUrl();
- LOG_DEBUG("Getting connection to broker: " << logicalAddress);
- const std::string& physicalAddress =
- data->shouldProxyThroughServiceUrl() ? serviceUrl_ : logicalAddress;
- Future<Result, ClientConnectionWeakPtr> future =
- pool_.getConnectionAsync(logicalAddress, physicalAddress);
- future.addListener(std::bind(&ClientImpl::handleNewConnection, shared_from_this(),
- std::placeholders::_1, std::placeholders::_2, promise));
- } else {
- promise.setFailed(result);
+ const auto topicNamePtr = TopicName::get(topic);
+ if (!topicNamePtr) {
+ LOG_ERROR("Unable to parse topic - " << topic);
+ promise.setFailed(ResultInvalidTopicName);
+ return promise.getFuture();
}
-}
-void ClientImpl::handleNewConnection(Result result, const ClientConnectionWeakPtr& conn,
- Promise<Result, ClientConnectionWeakPtr> promise) {
- if (result == ResultOk) {
- promise.setValue(conn);
- } else {
- promise.setFailed(ResultConnectError);
- }
+ auto self = shared_from_this();
+ lookupServicePtr_->getBroker(*topicNamePtr)
Review Comment:
We should pass a `std::shared_ptr` here, otherwise the object that `TopicNamePtr` points to might be invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#issuecomment-1229848378
@RobertIndie @merlimat @rdhabalia @gaoran10 Could any of you take a second look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#discussion_r958037731
##########
pulsar-client-cpp/lib/ClientImpl.cc:
##########
@@ -414,36 +402,32 @@ void ClientImpl::handleConsumerCreated(Result result, ConsumerImplBaseWeakPtr co
Future<Result, ClientConnectionWeakPtr> ClientImpl::getConnection(const std::string& topic) {
Promise<Result, ClientConnectionWeakPtr> promise;
- lookupServicePtr_->lookupAsync(topic).addListener(std::bind(&ClientImpl::handleLookup, shared_from_this(),
- std::placeholders::_1, std::placeholders::_2,
- promise));
- return promise.getFuture();
-}
-void ClientImpl::handleLookup(Result result, LookupDataResultPtr data,
- Promise<Result, ClientConnectionWeakPtr> promise) {
- if (data) {
- const std::string& logicalAddress =
- clientConfiguration_.isUseTls() ? data->getBrokerUrlTls() : data->getBrokerUrl();
- LOG_DEBUG("Getting connection to broker: " << logicalAddress);
- const std::string& physicalAddress =
- data->shouldProxyThroughServiceUrl() ? serviceUrl_ : logicalAddress;
- Future<Result, ClientConnectionWeakPtr> future =
- pool_.getConnectionAsync(logicalAddress, physicalAddress);
- future.addListener(std::bind(&ClientImpl::handleNewConnection, shared_from_this(),
- std::placeholders::_1, std::placeholders::_2, promise));
- } else {
- promise.setFailed(result);
+ const auto topicNamePtr = TopicName::get(topic);
+ if (!topicNamePtr) {
+ LOG_ERROR("Unable to parse topic - " << topic);
+ promise.setFailed(ResultInvalidTopicName);
+ return promise.getFuture();
}
-}
-void ClientImpl::handleNewConnection(Result result, const ClientConnectionWeakPtr& conn,
- Promise<Result, ClientConnectionWeakPtr> promise) {
- if (result == ResultOk) {
- promise.setValue(conn);
- } else {
- promise.setFailed(ResultConnectError);
- }
+ auto self = shared_from_this();
+ lookupServicePtr_->getBroker(*topicNamePtr)
Review Comment:
But it should be okay in this PR. Because only the returned `std::string` is used in `getBroker` method. But if we applied the retry logic in future, it would be a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17162: [feat][cpp] Support multiple brokers in service URL
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #17162:
URL: https://github.com/apache/pulsar/pull/17162#issuecomment-1663217922
See my comment https://github.com/apache/pulsar/pull/3249#issuecomment-1663216696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org