You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/07 13:19:17 UTC
[GitHub] [pulsar] ericsyh opened a new issue, #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
ericsyh opened a new issue, #15487:
URL: https://github.com/apache/pulsar/issues/15487
For a pulsar cluster with both proxy and broker running on istio with the support, I noticed an interesting problem that all http requests forwarded by pulsar proxies to brokers would fail, but other requests seem to be working fine, including
- pulsar binary requests that produce & consume messages
- http requests served by proxies directly, like `/status.html`, `/metrics/`
- http requests issued from the proxy pod directly using commands like `curl`
The error request and response looks just like
```
➜ ~ curl -v https://proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev/admin/v2/clusters
* Trying 198.18.9.176:443...
* Connected to proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev (198.18.9.176) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.sn-grizzly-dev-us-west-2.test.aws.sn2.dev
* start date: Feb 9 06:56:33 2022 GMT
* expire date: May 10 06:56:32 2022 GMT
* subjectAltName: host "proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev" matched cert's "*.sn-grizzly-dev-us-west-2.test.aws.sn2.dev"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
> GET /admin/v2/clusters HTTP/1.1
> Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Date: Thu, 17 Mar 2022 08:02:22 GMT
< Content-Type: text/plain
< Date: Thu, 17 Mar 2022 08:02:21 GMT
< Server: envoy
< Content-Length: 95
<
* Connection #0 to host proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination%
```
After some investigations, it seems the proxy would copy as much of the original http request as possible and send them to the broker service, as shown in the proxy debug log, including the original `HOST` value which might actually caused the issue:
```
2022-03-17T07:47:59,957+0000 [pulsar-external-web-5-4] DEBUG org.apache.pulsar.proxy.server.AdminProxyHandler.7c6189d5 - 1945615336 proxying to upstream:
GET /admin/v2/clusters HTTP/1.1
Accept: */*
User-Agent: curl/7.79.1
Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev
HttpRequest[GET /admin/v2/clusters HTTP/1.1]@4759582c
Accept: */*
User-Agent: curl/7.79.1
Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev
Via: 1.1 sn-sn-platform-proxy-0
X-Forwarded-For: 10.80.20.192
X-Forwarded-Proto: https
X-Forwarded-Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev
X-Forwarded-Server: 10.80.10.189
```
And the issue could be reproduced with `curl` on the proxy pod using the command `curl -v "$brokerWebServiceURL/admin/v2/clusters" -H 'HOST: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev'`, while `curl -v "$brokerWebServiceURL/admin/v2/clusters"` works as expected. On the other hand, specifying the broker hostname when making the http request to the proxy would also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15487:
URL: https://github.com/apache/pulsar/issues/15487#issuecomment-1150594213
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericsyh commented on issue #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
Posted by GitBox <gi...@apache.org>.
ericsyh commented on issue #15487:
URL: https://github.com/apache/pulsar/issues/15487#issuecomment-1120209279
Host header mismatch is the root cause, and proxy should strip HOST header. CC @fantapsody
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yuanmabiji commented on issue #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
Posted by GitBox <gi...@apache.org>.
yuanmabiji commented on issue #15487:
URL: https://github.com/apache/pulsar/issues/15487#issuecomment-1120385113
hello
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #15487: [bug][proxy]: http requests forwarded by pulsar proxies to brokers will fail when using Istio
URL: https://github.com/apache/pulsar/issues/15487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org