[GitHub] [pulsar] addisonj opened a new issue #5360: Proxy doesn't send request body on redirects

[GitBox <gi...@apache.org>]

addisonj opened a new issue #5360: Proxy doesn't send request body on redirects
URL: https://github.com/apache/pulsar/issues/5360
 
 
   **Describe the bug**
   When using multiple brokers behind a proxy, requests with a request body from the client fail. See #3702 as an example. The following seems to be what happens:
   
   - The client makes a request to proxy
   - The proxy forward the requests to the broker
   - The broker is not authoritative for that request, so it responds with a redirect
   - The proxy follows the redirect but DOES NOT forward on the original request body
   - The second broker, seeing a content-length header but never getting a body, waits for a full response but then times out
   - The proxy responds with a 504 gateway timeout
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. Run multiple brokers
   2. Run a proxy in front of the brokers
   3. make an admin request that takes a request body and must be routed to a specific broker, like creating a subscription to a specific ledger id (a default subscription doesn't send a request body)
   4. Try a few times, you should eventually get a gateway timeout
   
   This appears to be the case with and without TLS. I haven't yet tried it without authentication
   
   **Expected behavior**
   The request should succeed, with the request body being sent with the redirect
   
   **Screenshots**
   I validate this with tcp dumps:
   
   The initial request to the first broker:
   ```
   PUT /admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae HTTP/1.1
   Authorization: Bearer ...
   User-Agent: Pulsar-Java-v2.4.1
   Host: pulsar-pdx.bus-beta.insk8s.net:8443
   Accept: application/json
   Content-Type: application/json
   Via: http/1.1 pulsar-beta-proxy-6b87d7f8cc-thmmp
   X-Forwarded-For: 10.11.62.69
   X-Forwarded-Proto: https
   X-Forwarded-Host: pulsar-pdx.bus-beta.insk8s.net:8443
   X-Forwarded-Server: 10.11.61.62
   X-Original-Principal: code-admin
   Content-Length: 82
   
   {"ledgerId":9223372036854775807,"entryId":9223372036854775807,"partitionIndex":-1}
   
   HTTP/1.1 307 Temporary Redirect
   Date: Thu, 10 Oct 2019 23:11:41 GMT
   Location: http://10.11.41.134:8080/admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae?authoritative=false
   broker-address: 10.11.54.198
   Content-Length: 0
   Server: Jetty(9.4.12.v20180830)
   ```
   
   The redirect to the second broker (from the proxy)
   ```
   PUT /admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae?authoritative=false HTTP/1.1
   User-Agent: Jetty/9.4.12.v20180830
   User-Agent: Pulsar-Java-v2.4.1
   Accept: application/json
   Content-Type: application/json
   Via: http/1.1 pulsar-beta-proxy-6b87d7f8cc-thmmp
   X-Forwarded-For: 10.11.62.69
   X-Forwarded-Proto: https
   X-Forwarded-Host: pulsar-pdx.bus-beta.insk8s.net:8443
   X-Forwarded-Server: 10.11.61.62
   X-Original-Principal: code-admin
   Authorization: Bearer ...
   Host: 10.11.41.134:8080
   Content-Length: 82
   
   HTTP/1.1 500 Request failed.
   Date: Thu, 10 Oct 2019 23:11:41 GMT
   Content-Length: 0
   Server: Jetty(9.4.12.v20180830)
   ```
   
   Note that the content-length is set but no body is sent.
   
   
   **Desktop (please complete the following information):**
   Occurs with Pulsar 2.4.1 on linux
   
   **Additional context**
   I have spent a fair amount of time trying to debug this, here are some known details:
   
   In `AdminProxyHandler`, we copy the request, see https://github.com/apache/pulsar/blob/v2.4.1/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L152
   
   The super method here, https://github.com/eclipse/jetty.project/blob/jetty-9.4.12.v20180830/jetty-client/src/main/java/org/eclipse/jetty/client/HttpClient.java#L459, copies the content.
   
   However, the contentProvider may be an iterator, that depending on the implementation, can't be reset. It isn't clear what implementation would be used by the proxy. 
   
   Additionally, this is made more difficult as it seems like the admin proxy isn't logging as expected. With debug level logs turned on, I didn't see logs from the proxy for some code paths I am sure it should be in.
   
   I think the next step might be to have the AdminProxyHandler consume the request body before sending the first request and then re-attaching it for the first and any subsequent proxied requests
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services