You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/24 11:23:06 UTC
[GitHub] [pulsar-helm-chart] tabalt opened a new issue #25: Pods of broker/proxy/recovery init failed when enabled tls
tabalt opened a new issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25
**Describe the bug**
Pods of broker/proxy/recovery init failed when enabled tls
**To Reproduce**
Install commands：
```
git clone https://github.com/apache/pulsar-helm-chart.git ./
cd pulsar-helm-chart/
./scripts/cert-manager/install-cert-manager.sh
./scripts/pulsar/prepare_helm_release.sh -c -n pulsar -k pulsar
helm upgrade --install pulsar charts/pulsar \
--set namespace=pulsar --set volumes.local_storage=true --set certs.internal_issuer.enabled=true \
--set tls.enabled=true --set tls.proxy.enabled=true --set tls.broker.enabled=true --set tls.bookie.enabled=true \
--set tls.zookeeper.enabled=true --set tls.autorecovery.enabled=true --set tls.toolset.enabled=true \
--set auth.authentication.enabled=true --set auth.authorization.enabled=true -n pulsar
```
**Expected behavior**
Pods of broker/proxy/recovery stucked in the Init status
```
kubectl get pods -n pulsar
NAME READY STATUS RESTARTS AGE
pulsar-bookie-0 1/1 Running 0 46m
pulsar-bookie-1 1/1 Running 0 46m
pulsar-bookie-2 1/1 Running 0 46m
pulsar-bookie-3 1/1 Running 0 46m
pulsar-bookie-init-l9zdv 0/1 Completed 0 46m
pulsar-broker-0 0/1 Init:0/2 0 46m
pulsar-broker-1 0/1 Init:0/2 0 46m
pulsar-broker-2 0/1 Init:0/2 0 46m
pulsar-grafana-5ffd75b49d-g658b 1/1 Running 0 46m
pulsar-prometheus-5f957bf77-6mj2z 1/1 Running 0 46m
pulsar-proxy-0 0/1 Init:1/2 0 46m
pulsar-proxy-1 0/1 Init:1/2 0 46m
pulsar-proxy-2 0/1 Init:1/2 0 46m
pulsar-pulsar-init-mqsvt 1/1 Running 0 46m
pulsar-pulsar-manager-767d5f5766-khpr4 1/1 Running 0 46m
pulsar-recovery-0 0/1 Init:0/1 0 46m
pulsar-toolset-0 1/1 Running 0 46m
pulsar-zookeeper-0 1/1 Running 0 46m
pulsar-zookeeper-1 1/1 Running 0 46m
pulsar-zookeeper-2 1/1 Running 0 45m
```
Check file /pulsar/certs/broker/tls.crt failed when init container started
```
kubectl logs pulsar-broker-0 -c wait-zookeeper-ready -n pulsar | head -8
processing /pulsar/certs/broker/tls.crt : len = 0
/pulsar/certs/broker/tls.crt is empty
JMX enabled by default
Connecting to pulsar-zookeeper:2281
...
```
When I check it, tls files had generated
```
kubectl exec -it pulsar-broker-0 -c wait-zookeeper-ready -n pulsar /bin/bash
ls -al /pulsar/certs/broker/tls.crt
lrwxrwxrwx 1 root root 14 Jun 24 10:06 /pulsar/certs/broker/tls.crt -> ..data/tls.crt
``
If I re-run the following command：
```
/pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true;
```
The init container will be successful exit, and pod will running
```
kubectl get pods -n pulsar | grep 'pulsar-broker-0'
pulsar-broker-0 1/1 Running 0 71m
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie closed issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
sijie closed issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tabalt commented on issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
tabalt commented on issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25#issuecomment-649571662
I think move the `keytool.sh` after `until` as the following can solve the problem:
```
until /pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true && bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
But I don't know if there's a better way to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tabalt commented on issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
tabalt commented on issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25#issuecomment-649549983
@sijie Yes, I have installed cert-manager before installing the Pulsar helm chart.
The certificate files generated success when I check it, but it did’t generated when the pods of broker/proxy/recovery start. So the following init script execute failed:
```
/pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true;
```
And the following check script will be failed every times:
```
until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tabalt edited a comment on issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
tabalt edited a comment on issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25#issuecomment-649571662
The current script for init container like the following:
```
/pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true; until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
I think move the `keytool.sh` after `until` as the following can solve the problem:
```
until /pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true && bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
But I don't know if there's a better way to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie commented on issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25#issuecomment-648952940
@tabalt Did you install cert-manager before installing the Pulsar helm chart? The helm chart uses cert-manager to issue self-signed certificates.
See: http://pulsar.apache.org/docs/en/helm-deploy/#install-cert-manager
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tabalt removed a comment on issue #25: Pods of broker/proxy/recovery init failed when enabled tls
Posted by GitBox <gi...@apache.org>.
tabalt removed a comment on issue #25:
URL: https://github.com/apache/pulsar-helm-chart/issues/25#issuecomment-649571662
The current script for init container like the following:
```
/pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true; until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
I think move the `keytool.sh` after `until` as the following can solve the problem:
```
until /pulsar/keytool/keytool.sh broker ${HOSTNAME}.pulsar-broker.pulsar.svc.cluster.local true && bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server pulsar-zookeeper:2281 get /admin/clusters/pulsar; do echo "pulsar cluster pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3; done;
```
But I don't know if there's a better way to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org