You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/11/19 08:13:34 UTC
[GitHub] [pulsar] wolfstudy opened a new issue #12888: CPP SDK did not respond to request 307 during lookup
wolfstudy opened a new issue #12888:
URL: https://github.com/apache/pulsar/issues/12888
**Describe the bug**
A normal broker cluster, suppose I now create a topic with 128 partitions to ensure that each broker node is distributed with a certain number of partitions. At this time, start the cpp client to send and consume data, everything is normal.
At this point, we kill one of the brokers and restart the broker. We can see that the cpp sdk keeps reporting the following error
```
2021-11-19 13:39:08.838 ERROR [140028414940928] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-82. Error Code 22
2021-11-19 13:39:13.220 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-16. Error Code 22
2021-11-19 13:39:14.291 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-50. Error Code 22
2021-11-19 13:39:15.372 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-54. Error Code 22
2021-11-19 13:39:16.451 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-58. Error Code 22
2021-11-19 13:39:18.497 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-33. Error Code 22
2021-11-19 13:39:19.238 ERROR [140028414940928] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-88. Error Code 22
2021-11-19 13:39:19.567 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-24. Error Code 22
2021-11-19 13:39:20.343 ERROR [140028414940928] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-89. Error Code 22
2021-11-19 13:39:21.653 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-13. Error Code 22
2021-11-19 13:39:22.722 ERROR [140028406548224] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-12. Error Code 22
2021-11-19 13:39:23.040 ERROR [140028414940928] HTTPLookupService:262 | Response failed for url http://pulsar-q4822m9k7ej8.tdmq.ap-ru.public.tencenttdmq.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-91. Error Code 22
```
Since it is using HTTP Lookup service, it uses the libcurl warehouse. At this time, we use the curl command locally to simulate the processing behavior of the cpp code:
```
curl -i http://pulsar-q4822m9k7ej8.ap-public.com:8080/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-33 -H 'Authorization: Bearer eyJrZXlJZCI6Ixxxxxx'
```
We got the following wrong output:
At this point, we continue to use curl to execute the Location returned last time, as follows:
```
curl -i http://1.62.3.1:1001/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-30 -H 'Authorization: Bearer eyJrZXlJZCI6InB1bHNhci1xcccccxxxxx'
```
Output as follows:
```
{"brokerUrl":"pulsar://1.14.81.60:6650","httpUrl":"http://1.14.81.60:8080","nativeUrl":"pulsar://1.14.81.60:6650","netAddressJson":"pulsar://1.14.81.60:6650:6650","brokerUrlSsl":""}
```
Through the return value, we can see that the broker returned the correct owner broker address to the cpp client, but the cpp sdk did not respond to the 307 request, causing the lookup to continue to fail.
**Desktop (please complete the following information):**
- CPP SDK version: 2.7.2
- Broker version: 2.7.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-973906020
It looks like the C++ client has configured the URL redirection, see https://github.com/apache/pulsar/blob/b807200552117c299e3776e33537d082a00290d5/pulsar-client-cpp/lib/HTTPLookupService.cc#L182-L184
And it's weird that you log is
```
Error Code 22
```
See https://curl.se/libcurl/c/libcurl-errors.html
> CURLE_HTTP_RETURNED_ERROR (22)
>
> This is returned if CURLOPT_FAILONERROR is set TRUE and the HTTP server returns an error code that is >= 400.
Could you try following C++ code and see what would happen?
```c++
#include <curl/curl.h>
#include <iostream>
#include <string>
static size_t curlWriteCallback(void *contents, size_t size, size_t nmemb,
void *responseDataPtr) {
((std::string *)responseDataPtr)->append((char *)contents, size * nmemb);
return size * nmemb;
}
int main() {
std::string completeUrl =
"http://1.62.3.1:1001/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-30";
std::string header =
"Authorization: Bearer eyJrZXlJZCI6InB1bHNhci1xcccccxxxxx";
CURL *handle = curl_easy_init();
if (!handle) {
return 1;
}
// set URL
curl_easy_setopt(handle, CURLOPT_URL, completeUrl.c_str());
// Write callback
std::string responseData;
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, curlWriteCallback);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, &responseData);
// New connection is made for each call
curl_easy_setopt(handle, CURLOPT_FRESH_CONNECT, 1L);
curl_easy_setopt(handle, CURLOPT_FORBID_REUSE, 1L);
// Skipping signal handling - results in timeouts not honored during the DNS
// lookup
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
// Timer
curl_easy_setopt(handle, CURLOPT_TIMEOUT, 3);
// Set User Agent
curl_easy_setopt(handle, CURLOPT_USERAGENT, "Pulsar-CPP-v2.8.0");
// Redirects
curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(handle, CURLOPT_MAXREDIRS, 20);
// Fail if HTTP return code >=400
curl_easy_setopt(handle, CURLOPT_FAILONERROR, 1L);
struct curl_slist *list = NULL;
list = curl_slist_append(list, header.c_str());
curl_easy_setopt(handle, CURLOPT_HTTPHEADER, list);
char errorBuffer[CURL_ERROR_SIZE];
curl_easy_setopt(handle, CURLOPT_ERRORBUFFER, errorBuffer);
curl_easy_setopt(handle, CURLOPT_CUSTOMREQUEST, "GET");
auto res = curl_easy_perform(handle);
std::cout << "Result: " << res << ", error: " << errorBuffer << std::endl;
long httpResponseCode;
curl_easy_getinfo(handle, CURLINFO_RESPONSE_CODE, &httpResponseCode);
std::cout << "HTTP response code: " << httpResponseCode << std::endl;
curl_easy_cleanup(handle);
std::cout << "Content:\n" << responseData << std::endl;
return 0;
}
```
You need to modify `completeUrl` and `header` according to your URL.
For example, in my local env, the output could be
```
Result: 0, error:
HTTP response code: 200
Content:
{"brokerUrl":"pulsar://localhost:6650","httpUrl":"http://localhost:8080","nativeUrl":"pulsar://localhost:6650","brokerUrlSsl":""}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-975433643
In here, i try to run `tcpdump` and get the follow output:
```
19:28:48.231477 IP 62.6.3.204.webcache > VM-14-64-centos.48860: Flags [P.], seq 568194484:568195512, ack 2190479487, win 122, options [nop,nop,TS val 729815901 ecr 1215747908], length 1028: HTTP: HTTP/1.1 307 Temporary Redirect
E..8.W@.@....>.. ..@....!..........z.......
+..]Hv.DHTTP/1.1 307 Temporary Redirect
Date: Mon, 22 Nov 2021 11:28:48 GMT
Content-Type: text/html;charset=iso-8859-1
Content-Length: 598
Connection: keep-alive
broker-address: 1.1.5.60
Location: http://12.6.3.204:5001/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-63?authoritative=true&listenerName=PUBLIC
Cache-Control: must-revalidate,no-cache,no-store
Server: Jetty(9.4.43.v20210629)
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 307 Temporary Redirect</title>
</head>
<body><h2>HTTP ERROR 307 Temporary Redirect</h2>
<table>
<tr><th>URI:</th><td>/lookup/v2/topic/persistent/pulsar-q4822m9k7ej8/test-ns/test-topic-1-partition-63</td></tr>
<tr><th>STATUS:</th><td>307</td></tr>
<tr><th>MESSAGE:</th><td>Temporary Redirect</td></tr>
<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-22d9ca63</td></tr>
</table>
<hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.43.v20210629</a><hr/>
</body>
</html>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy edited a comment on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
wolfstudy edited a comment on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-974584546
When i run the code, return `403 Forbidden`
@BewareMyPower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-973852386
ping @BewareMyPower PTAL thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-974584546
When i run the code, return `403 Forbidden`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] billowqiu commented on issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
billowqiu commented on issue #12888:
URL: https://github.com/apache/pulsar/issues/12888#issuecomment-985272090
When restart one broker，i use tcpdump to capture packet，it show that libcurl follow new url without auth token header
<img width="1405" alt="wecom-temp-734f584adac9a9f900e67e1ff04289ea" src="https://user-images.githubusercontent.com/720496/144560905-8845ff3c-ed71-457b-a719-abda50626978.png">
<img width="1402" alt="wecom-temp-38c19b01c3bd6bba29650e0f6dba76df" src="https://user-images.githubusercontent.com/720496/144560942-8c5171d9-7a73-428a-b068-eae301c9982e.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 closed issue #12888: CPP SDK did not respond to request 307 during lookup
Posted by GitBox <gi...@apache.org>.
315157973 closed issue #12888:
URL: https://github.com/apache/pulsar/issues/12888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org