[GitHub] [pulsar] one70six opened a new issue #4612: Pulsar proxy not routing https requests to function workers

[GitBox <gi...@apache.org>]

one70six opened a new issue #4612: Pulsar proxy not routing https requests to function workers
URL: https://github.com/apache/pulsar/issues/4612
 
 
   **Describe the bug**
   Pulsar v2.3.2
   
   After following the Pulsar documentation from the following link, [Configure Proxies for Functions-workers](https://pulsar.apache.org/docs/en/functions-worker/#configure-proxies-for-functions-workers), to run functions workers on separate machines outside the brokers, any pulsar-admin cli request to functions results in the following error:
   
   `Function worker service is not done initializing. Please try again in a little while.
   Reason: HTTP 503 Service Unavailable`
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. Start with a TLS enabled cluster that has brokers and pulsar proxies
   2. client.conf points to the proxy.conf webServicePortTls so that all admin requests go through 
   the proxies
   3. This means the default admin-url == the pulsar proxy url and port  for webServicePortTls 
   4. My proxy.conf setup is as follows:
   
   **proxy.conf**
   ```bash
   ### --- Broker Discovery --- ###
   
   # The ZooKeeper quorum connection string (as a comma-separated list)
   zookeeperServers=zk1:2181,zk2:2181,zk3:2181
   
   # Configuration store connection string (as a comma-separated list)
   configurationStoreServers=zk1:2181,zk2:2181,zk3:2181
   
   # if Service Discovery is Disabled this url should point to the discovery service provider.
   brokerServiceURL=
   brokerServiceURLTLS=
   
   # These settings are unnecessary if `zookeeperServers` is specified
   brokerWebServiceURL=
   brokerWebServiceURLTLS=
   
   # If function workers are setup in a separate cluster, configure the following 2 settings
   # to point to the function workers cluster
   functionWorkerWebServiceURL=http://myfunctionscluster:6750
   functionWorkerWebServiceURLTLS=https://myfunctionscluster:6751
   
   # ZooKeeper session timeout (in milliseconds)
   zookeeperSessionTimeoutMs=30000
   
   ### --- Server --- ###
   
   # The port to use for server binary Protobuf requests
   servicePort=16650
   
   # The port to use to server binary Protobuf TLS requests
   servicePortTls=16651
   
   # Port that discovery service listen on
   webServicePort=18080
   
   # Port to use to server HTTPS request
   webServicePortTls=18443
   
   # Path for the file used to determine the rotation status for the proxy instance when responding
   # to service discovery health checks
   statusFilePath=
   
   ### ---Authorization --- ###
   
   # Role names that are treated as "super-users," meaning that they will be able to perform all admin
   # operations and publish/consume to/from all topics (as a comma-separated list)
   superUserRoles=admin
   
   # Whether authorization is enforced by the Pulsar proxy
   authorizationEnabled=false
   
   # Authorization provider as a fully qualified class name
   authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
   
   # Whether client authorization credentials are forwared to the broker for re-authorization.
   # Authentication must be enabled via authenticationEnabled=true for this to take effect.
   forwardAuthorizationCredentials=true
   
   ### --- Authentication --- ###
   
   # Whether authentication is enabled for the Pulsar proxy
   authenticationEnabled=true
   
   # Authentication provider name list (a comma-separated list of class names)
   authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
   
   ### --- Client Authentication --- ###
   
   # The three brokerClient* authentication settings below are for the proxy itself and determine how it
   # authenticates with Pulsar brokers
   
   # The authentication plugin used by the Pulsar proxy to authenticate with Pulsar brokers
   brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
   
   # The authentication parameters used by the Pulsar proxy to authenticate with Pulsar brokers
   brokerClientAuthenticationParameters=tlsCertFile:/opt/pulsar/certificates/proxy.cert.pem,tlsKeyFile:/opt/pulsar/certificates/proxy.key-pk8.pem
   
   # The path to trusted certificates used by the Pulsar proxy to authenticate with Pulsar brokers
   brokerClientTrustCertsFilePath=/opt/pulsar/certificates/ca.cert.pem
   
   # Whether TLS is enabled when communicating with Pulsar brokers
   tlsEnabledWithBroker=true
   
   ##### --- Rate Limiting --- #####
   
   # Max concurrent inbound connections. The proxy will reject requests beyond that.
   maxConcurrentInboundConnections=10000
   
   # Max concurrent outbound connections. The proxy will error out requests beyond that.
   maxConcurrentLookupRequests=50000
   
   ##### --- TLS --- #####
   
   # Deprecated - use servicePortTls and webServicePortTls instead
   tlsEnabledInProxy=true
   
   # Path for the TLS certificate file
   tlsCertificateFilePath=/opt/pulsar/certificates/broker.cert.pem
   
   # Path for the TLS private key file
   tlsKeyFilePath=/opt/pulsar/certificates/broker.key-pk8.pem
   
   # Path for the trusted TLS certificate file.
   # This cert is used to verify that any certs presented by connecting clients
   # are signed by a certificate authority. If this verification
   # fails, then the certs are untrusted and the connections are dropped.
   tlsTrustCertsFilePath=/opt/pulsar/certificates/ca.cert.pem
   
   # Accept untrusted TLS certificate from client.
   # If true, a client with a cert which cannot be verified with the
   # 'tlsTrustCertsFilePath' cert will allowed to connect to the server,
   # though the cert will not be used for client authentication.
   tlsAllowInsecureConnection=false
   
   # Whether the hostname is validated when the proxy creates a TLS connection with brokers
   tlsHostnameVerificationEnabled=false
   
   # Specify the tls protocols the broker will use to negotiate during TLS handshake
   # (a comma-separated list of protocol names).
   # Examples:- [TLSv1.2, TLSv1.1, TLSv1]
   tlsProtocols=TLSv1.2
   
   # Specify the tls cipher the broker will use to negotiate during TLS Handshake
   # (a comma-separated list of ciphers).
   # Examples:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
   tlsCiphers=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   
   # Whether client certificates are required for TLS. Connections are rejected if the client
   # certificate isn't trusted.
   tlsRequireTrustedClientCertOnConnect=true
   
   ##### --- HTTP --- #####
   
   # Http directs to redirect to non-pulsar services.
   httpReverseProxyConfigs=
   
   # Http output buffer size. The amount of data that will be buffered for http requests
   # before it is flushed to the channel. A larger buffer size may result in higher http throughput
   # though it may take longer for the client to see data.
   # If using HTTP streaming via the reverse proxy, this should be set to the minimum value, 1,
   # so that clients see the data as soon as possible.
   httpOutputBufferSize=32768
   
   # Number of threads to use for HTTP requests processing. Default is
   # 2 * Runtime.getRuntime().availableProcessors()
   httpNumThreads=
   
   ### --- Token Authentication Provider --- ###
   
   ## Symmetric key
   # Configure the secret key to be used to validate auth tokens
   # The key can be specified like:
   # tokenSecretKey=data:base64,xxxxxxxxx
   # tokenSecretKey=file:///my/secret.key
   tokenSecretKey=
   
   ## Asymmetric public/private key pair
   # Configure the public key to be used to validate auth tokens
   # The key can be specified like:
   # tokenPublicKey=data:base64,xxxxxxxxx
   # tokenPublicKey=file:///my/public.key
   tokenPublicKey=
   
   # The token "claim" that will be interpreted as the authentication "role" or "principal" by AuthenticationProviderToken (defaults to "sub" if blank)
   tokenAuthClaim=
   
   ### --- Deprecated config variables --- ###
   
   # Deprecated. Use configurationStoreServers
   globalZookeeperServers=
   ```
   
   **Expected behavior**
   The proxy should inspect the request from the admin cli, determine if it is for functions or other, and forward accordingly to functions worker cluster or brokers, etc.. 
   
   - I believe that is how the diagram is shown in the [Configure Proxies for Functions-workers](https://pulsar.apache.org/docs/en/functions-worker/#configure-proxies-for-functions-workers) link?
   - Possibly this bit of code is attempting to do just that?: [AdminProxyHandler](https://github.com/apache/pulsar/blob/v2.3.2/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java)
   
   **Additional context**
   You can force the correct behavior by manually overriding the `--admin-url` parameter on the pulsar-admin cli as follows before you run it. I was under the impression that if you set it up correctly, this was unnecessary?
   ```bash
   $PULSAR_HOME/bin/pulsar-admin \
   --admin-url  https://myfunctionscluster.com:6751 \
   functions getstatus \
   --tenant public \
   --namespace default \
   --name exclamation 
   ```
   
   Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services