You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/12 19:10:19 UTC
[GitHub] [pulsar-helm-chart] devinbost opened a new issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
devinbost opened a new issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84
Copying from the Apache/Pulsar Github issue (https://github.com/apache/pulsar/issues/8536):
**Describe the bug**
After configuring TLS Authentication in Pulsar 2.6.1 with this helm chart: https://github.com/devinbost/pulsar-helm-chart/tree/tls-auth
the broker gets stuck in a restart loop due to the `WorkerService` crashing with:
> WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
during the `WorkerService.start(..)` method execution.
With TLS Authentication enabled, the endpoint above should be the TLS endpoint (https://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8443/admin/v2/persistent/public/functions/assignments), not the non-TLS endpoint. This may be the reason why we're getting a 401 on the PUT for function/assignments upon the broker startup.
**To Reproduce**
Steps to reproduce the behavior:
1. Clone the tls-auth branch of my fork of the Pulsar helm chart by running:
```
git clone https://github.com/devinbost/pulsar-helm-chart.git
git checkout tls-auth
```
2. Start minikube with an appropriate number of CPUs:
minikube start --memory=8192 --cpus=6 --cni=bridge
3. Run the following commands to setup the kubernetes environment, tokens, certs, and keys:
```
./scripts/cert-manager/install-cert-manager.sh
./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-ci -c --pulsar-superusers superadmin,proxy-admin,broker-admin,client-admin
./scripts/pulsar/upload_tls.sh -k pulsar-ci -d ./.ci/tls
```
4. Install the local helm chart with the values file specified:
`helm install --values examples/values-minikube-with-tls-and-jwt.yaml pulsar-ci ./charts/pulsar/`
5. After waiting for a time, get logs from the broker:
`kubectl -n pulsar logs pulsar-ci-broker-0`
The logs should demonstrate the problem.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment**
- minikube v1.14.2 on Darwin 10.15.7
- Kubernetes v1.19.2 on Docker 19.03.8 ...
- Enabled addons: storage-provisioner, default-storageclass
- kubectl is configured to use "minikube"
**Additional Context**
Here is the WorkerConfig provided to the WorkerService, as reported in the logs:
```
01:07:20.757 [main] INFO org.apache.pulsar.functions.worker.WorkerService - Worker Configs: {
"workerId" : "c-pulsar-ci-fw-pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local-8080",
"workerHostname" : "pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local",
"workerPort" : 8080,
"workerPortTls" : 6751,
"authenticateMetricsEndpoint" : true,
"includeStandardPrometheusMetrics" : false,
"jvmGCMetricsLoggerClassName" : null,
"numHttpServerThreads" : 8,
"configurationStoreServers" : "pulsar-ci-zookeeper:2281",
"zooKeeperSessionTimeoutMillis" : 30000,
"zooKeeperOperationTimeoutSeconds" : 30,
"zooKeeperCacheExpirySeconds" : 300,
"connectorsDirectory" : "./connectors",
"narExtractionDirectory" : "/tmp",
"validateConnectorConfig" : false,
"functionsDirectory" : "./functions",
"functionMetadataTopicName" : "metadata",
"functionWebServiceUrl" : null,
"pulsarServiceUrl" : "pulsar://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:6650",
"pulsarWebServiceUrl" : "http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080",
"clusterCoordinationTopicName" : "coordinate",
"pulsarFunctionsNamespace" : "public/functions",
"pulsarFunctionsCluster" : "pulsar-ci",
"numFunctionPackageReplicas" : 1,
"downloadDirectory" : "download/pulsar_functions",
"stateStorageServiceUrl" : null,
"functionAssignmentTopicName" : "assignments",
"schedulerClassName" : "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler",
"failureCheckFreqMs" : 30000,
"rescheduleTimeoutMs" : 60000,
"initialBrokerReconnectMaxRetries" : 60,
"assignmentWriteMaxRetries" : 60,
"instanceLivenessCheckFreqMs" : 30000,
"clientAuthenticationPlugin" : "org.apache.pulsar.client.impl.auth.AuthenticationTls",
"clientAuthenticationParameters" : "tlsCertFile:/pulsar/certs/broker/tls.crt,tlsKeyFile:/pulsar/certs/broker/tls.key",
"bookkeeperClientAuthenticationPlugin" : null,
"bookkeeperClientAuthenticationParametersName" : null,
"bookkeeperClientAuthenticationParameters" : null,
"topicCompactionFrequencySec" : 1800,
"tlsEnabled" : true,
"tlsCertificateFilePath" : null,
"tlsKeyFilePath" : null,
"tlsTrustCertsFilePath" : "/pulsar/certs/ca/ca.crt",
"tlsAllowInsecureConnection" : false,
"tlsRequireTrustedClientCertOnConnect" : false,
"useTls" : false,
"tlsHostnameVerificationEnable" : false,
"tlsCertRefreshCheckDurationSec" : 300,
"authenticationEnabled" : true,
"authenticationProviders" : [ "org.apache.pulsar.broker.authentication.AuthenticationProviderToken", "org.apache.pulsar.broker.authentication.AuthenticationProviderTls" ],
"authorizationEnabled" : true,
"authorizationProvider" : "org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider",
"superUserRoles" : [ "broker-admin", "client-admin", "proxy-admin" ],
"properties" : { },
"brokerClientTrustCertsFilePath" : null,
"functionRuntimeFactoryClassName" : "org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory",
"functionRuntimeFactoryConfigs" : {
"changeConfigMap" : "pulsar-ci-functions-worker-config",
"changeConfigMapNamespace" : "pulsar",
"expectedMetricsCollectionInterval" : "30",
"extraFunctionDependenciesDir" : null,
"installUserCodeDependencies" : "true",
"javaInstanceJarLocation" : null,
"jobNamespace" : "pulsar",
"logDirectory" : "/tmp",
"pulsarAdminUrl" : "https://pulsar-ci-broker:8443/",
"pulsarDockerImageName" : "apachepulsar/pulsar:2.6.1",
"pulsarRootDir" : "/pulsar",
"pulsarServiceUrl" : "pulsar+ssl://pulsar-ci-broker:6651/",
"pythonInstanceLocation" : null,
"submittingInsidePod" : "true"
},
"secretsProviderConfiguratorClassName" : null,
"secretsProviderConfiguratorConfig" : null,
"functionInstanceMinResources" : null,
"functionAuthProviderClassName" : "org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider",
"runtimeCustomizerClassName" : null,
"runtimeCustomizerConfig" : { },
"maxPendingAsyncRequests" : 1000,
"threadContainerFactory" : null,
"processContainerFactory" : null,
"kubernetesContainerFactory" : {
"k8Uri" : null,
"jobNamespace" : "pulsar",
"pulsarDockerImageName" : "apachepulsar/pulsar:2.6.1",
"imagePullPolicy" : null,
"pulsarRootDir" : "/pulsar",
"configAdminCLI" : null,
"submittingInsidePod" : true,
"pulsarServiceUrl" : "pulsar+ssl://pulsar-ci-broker:6651/",
"pulsarAdminUrl" : "https://pulsar-ci-broker:8443/",
"installUserCodeDependencies" : true,
"pythonDependencyRepository" : null,
"pythonExtraDependencyRepository" : null,
"extraFunctionDependenciesDir" : null,
"customLabels" : null,
"expectedMetricsCollectionInterval" : 30,
"changeConfigMap" : "pulsar-ci-functions-worker-config",
"changeConfigMapNamespace" : "pulsar",
"percentMemoryPadding" : 0,
"cpuOverCommitRatio" : 1.0,
"memoryOverCommitRatio" : 1.0,
"grpcPort" : 9093,
"metricsPort" : 9094,
"narExtractionDirectory" : "/tmp"
},
"functionMetadataTopic" : "persistent://public/functions/metadata",
"clusterCoordinationTopic" : "persistent://public/functions/coordinate",
"functionAssignmentTopic" : "persistent://public/functions/assignments",
"tlsTrustChainBytes" : "LS0tLS1C. . . =",
"workerWebAddress" : "http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080"
}
```
**Clues and Possible Solution**
The only admin endpoints in the `WorkerConfig` that are NOT TLS are:
- "workerWebAddress" : "http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080"
- "pulsarWebServiceUrl" : "http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080"
- "functionWebServiceUrl" : null
When we create the `brokerAdmin` client, we use the `pulsarWebServiceUrl`: https://github.com/apache/pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java#L146
The first PUT on the function assignment topic uses the `brokerAdmin`client here: https://github.com/apache/pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java#L169
Although we check for a few TLS-related configurations (tlsTrustCertsFilePath, allowTlsInsecureConnection, enableTlsHostnameVerificationEnable) when we create the PulsarAdmin client, it doesn't appear that we ever resolve to obtain a TLS endpoint if TLS Authentication is enabled.
If a TLS endpoint is required to resolve the 401 response issue, we need to add logic to check if TLS Authentication is enabled; and when TLS Authentication is enabled, we need to use a TLS endpoint when creating the `AdminClient` instances, such as `brokerAdmin`.
We could easily add the logic to resolve the correct URL to the brokerConfig class (`ServiceConfiguration`) since this class already knows if `brokerClientTlsEnabled` is true/false: https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L1656
Then, that value could be assigned to a property on `workerConfig` before injecting `workerConfig` into our `WorkerService`: https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L175
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-845531235
@gubespam I ended up putting this on the shelf to work on higher priority items, but I suspect it's a configuration issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729301860
It appears (from the debug logs) that the TLS session is established.
The exception "Client unable to authenticate with TLS certificate" is thrown in AuthenticationProviderTls from this block:
@Override
public String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
String commonName = null;
if (authData.hasDataFromTls()) {
Certificate[] certs = authData.getTlsCertificates();
String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName();
for (String keyValueStr : distinguishedName.split(",")) {
String[] keyValue = keyValueStr.split("=", 2);
if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) {
commonName = keyValue[1];
break;
}
}
}
if (commonName == null) {
throw new AuthenticationException("Client unable to authenticate with TLS certificate");
}
return commonName;
}
That implies that the CN is blank... However, the TLS logs (see attached) show that a CN is clearly present.
So, I'm not sure that I understand what is wrong here.
[pulsarbroker.txt](https://github.com/apache/pulsar-helm-chart/files/5557017/pulsarbroker.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost edited a comment on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729301860
It appears (from the debug logs) that the TLS session is established.
The exception "Client unable to authenticate with TLS certificate" is thrown in AuthenticationProviderTls from this block:
@Override
public String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
String commonName = null;
if (authData.hasDataFromTls()) {
Certificate[] certs = authData.getTlsCertificates();
String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName();
for (String keyValueStr : distinguishedName.split(",")) {
String[] keyValue = keyValueStr.split("=", 2);
if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) {
commonName = keyValue[1];
break;
}
}
}
if (commonName == null) {
throw new AuthenticationException("Client unable to authenticate with TLS certificate");
}
return commonName;
}
(https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderTls.java#L86)
That implies that the CN is blank... However, the TLS logs (see attached) show that a CN is clearly present.
So, I'm not sure that I understand what is wrong here.
[pulsarbroker.txt](https://github.com/apache/pulsar-helm-chart/files/5557017/pulsarbroker.txt)
Edit: There's a more recent log in a later comment that shows the CNs matching superAdmin role names. Even after making that change, I'm still getting the same 401.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost edited a comment on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729301860
It appears (from the debug logs) that the TLS session is established.
The exception "Client unable to authenticate with TLS certificate" is thrown in AuthenticationProviderTls from this block:
@Override
public String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
String commonName = null;
if (authData.hasDataFromTls()) {
Certificate[] certs = authData.getTlsCertificates();
String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName();
for (String keyValueStr : distinguishedName.split(",")) {
String[] keyValue = keyValueStr.split("=", 2);
if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) {
commonName = keyValue[1];
break;
}
}
}
if (commonName == null) {
throw new AuthenticationException("Client unable to authenticate with TLS certificate");
}
return commonName;
}
(https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderTls.java#L86)
That implies that the CN is blank... However, the TLS logs (see attached) show that a CN is clearly present.
So, I'm not sure that I understand what is wrong here.
[pulsarbroker.txt](https://github.com/apache/pulsar-helm-chart/files/5557017/pulsarbroker.txt)
**Edit:** There's a more recent log in a later comment that shows the CNs matching superAdmin role names. Even after making that change, I'm still getting the same 401.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729305401
Just before the exception is thrown, it appears that the broker is successfully able to establish a TLS session with Zookeeper, but then it gives this odd message:
```
Inaccessible trust store: /usr/local/openjdk-8/jre/lib/security/jssecacerts
trustStore is: /usr/local/openjdk-8/jre/lib/security/cacerts
trustStore type is: jks
trustStore provider is:
the last modified time is: Thu Apr 16 10:21:14 UTC 2020
Reload the trust store
Reload trust certs
Reloaded 128 trust certs
```
and then loads a lot of certs, like:
```
adding as trusted cert:
Subject: CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK
Issuer: CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK
Algorithm: RSA; Serial number: 0x3e8
Valid from Thu May 15 05:13:14 UTC 2003 until Mon May 15 04:52:29 UTC 2023
adding as trusted cert:
Subject: CN=SecureTrust CA, O=SecureTrust Corporation, C=US
Issuer: CN=SecureTrust CA, O=SecureTrust Corporation, C=US
Algorithm: RSA; Serial number: 0xcf08e5c0816a5ad427ff0eb271859d0
Valid from Tue Nov 07 19:31:18 UTC 2006 until Mon Dec 31 19:40:55 UTC 2029
. . .
```
Immediately after it loads those certs, it reports:
```
trigger seeding of SecureRandom
done seeding SecureRandom
```
and then gets the 401 with:
`org.apache.pulsar.broker.web.AuthenticationFilter - [10.244.0.9] Failed to authenticate HTTP request: Client unable to authenticate with TLS certificate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-732387678
Here's a complete broker log.
[brokerlogs.txt](https://github.com/apache/pulsar-helm-chart/files/5585412/brokerlogs.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729225278
I added TLS debugging. (I forgot to tag this issue in the commit.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-731392632
I tried changing all the certs to use CNs that match roles specified as superAdmin roles, but I can't get beyond the exception:
```
20:26:40.155 [AsyncHttpClient-64-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-ci-broker-0.pulsar-ci-broker.pulsar.svc.cluster.local:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
20:26:40.173 [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: HTTP 401 Unauthorized
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] gubespam commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
gubespam commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-845474298
@devinbost Did you ever find a solution for this? I am running to the same problem you described in https://github.com/apache/pulsar/issues/8536 . It seems to me to be related to how the function worker is connecting to the broker, but it doesn't have anything to do with the helm chart itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] vitosans edited a comment on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
vitosans edited a comment on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-932431670
I did something like this:
```
webServiceUrl: "https://{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}:{{ .Values.proxy.ports.https }}/"
brokerServiceUrl: "pulsar+ssl://{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}:{{ .Values.proxy.ports.pulsarssl }}/"
tlsEnabled: "true"
brokerClientTlsEnabled: "true"
brokerClientTrustCertsFilePath: "/pulsar/certs/ca/ca.crt"
useTls: true
tlsCertificateFilePath: "/pulsar/certs/broker/tls.crt"
tlsKeyFilePath: "/pulsar/certs/broker/tls.key"
tlsTrustCertsFilePath: "/pulsar/certs/ca/ca.crt"
tlsAllowInsecureConnection: false
tlsEnableHostnameVerification: false
tlsCertRefreshCheckDurationSec: 300
```
In broker-configmap.yaml
The broker is now able to start up when functions are enabled. Now the problem is when you deploy a function the functions_worker that gets spawned off has a default functions_works.yaml and not the one generated from bin/gen-yml-from-env.py conf/functions_worker.yml in the StateFullSet
So of course he now gets a:
HTTP 401 Unauthorized │
Reason: HTTP 401 Unauthorized
as he is trying to post to http://localhost:8080 which of course is wrong :)
Trying to debug this currently, and then make a giant PR that enables mTLS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] vitosans commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
vitosans commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-932431670
I did something like this:
tlsEnabled: "true"
brokerClientTlsEnabled: "true"
brokerClientTrustCertsFilePath: "/pulsar/certs/ca/ca.crt"
useTls: true
tlsCertificateFilePath: "/pulsar/certs/broker/tls.crt"
tlsKeyFilePath: "/pulsar/certs/broker/tls.key"
tlsTrustCertsFilePath: "/pulsar/certs/ca/ca.crt"
tlsAllowInsecureConnection: false
tlsEnableHostnameVerification: false
tlsCertRefreshCheckDurationSec: 300
In broker-configmap.yaml
The broker is now able to start up when functions are enabled. Now the problem is when you deploy a function the functions_worker that gets spawned off has a default functions_works.yaml and not the one generated from bin/gen-yml-from-env.py conf/functions_worker.yml in the StateFullSet
So of course he now gets a:
HTTP 401 Unauthorized │
Reason: HTTP 401 Unauthorized
as he is trying to post to http://localhost:8080 which of course is wrong :)
Trying to debug this currently, and then make a giant PR that enables mTLS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost commented on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-726350229
After I swapped out the brokerClient auth to use token auth instead of TLS auth in the broker-configmap.yaml and proxy-configmap.yaml files, the cluster started just fine. So, it seems that there's a problem in the broker client TLS auth.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] devinbost edited a comment on issue #84: TLS Authentication in Kubernetes, Pulsar 2.6.1 - Broker crash loop on startup due to 401 in WorkerService.start(..)
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on issue #84:
URL: https://github.com/apache/pulsar-helm-chart/issues/84#issuecomment-729301860
It appears (from the debug logs) that the TLS session is established.
The exception "Client unable to authenticate with TLS certificate" is thrown in AuthenticationProviderTls from this block:
@Override
public String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
String commonName = null;
if (authData.hasDataFromTls()) {
Certificate[] certs = authData.getTlsCertificates();
String distinguishedName = ((X509Certificate) certs[0]).getSubjectX500Principal().getName();
for (String keyValueStr : distinguishedName.split(",")) {
String[] keyValue = keyValueStr.split("=", 2);
if (keyValue.length == 2 && "CN".equals(keyValue[0]) && !keyValue[1].isEmpty()) {
commonName = keyValue[1];
break;
}
}
}
if (commonName == null) {
throw new AuthenticationException("Client unable to authenticate with TLS certificate");
}
return commonName;
}
(https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderTls.java#L86)
That implies that the CN is blank... However, the TLS logs (see attached) show that a CN is clearly present.
So, I'm not sure that I understand what is wrong here.
[pulsarbroker.txt](https://github.com/apache/pulsar-helm-chart/files/5557017/pulsarbroker.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org