You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/11 13:11:13 UTC
[GitHub] [pulsar] cdbartholomew opened a new issue #7243: Bookies failing to start with Helm chart
cdbartholomew opened a new issue #7243:
URL: https://github.com/apache/pulsar/issues/7243
**Describe the bug**
The 2.6.0 candidate load cannot be upgraded using the project Helm chart. After upgrading, the bookies will not start because the startup parameters are invalid for the init container. This is the error message seen in the pulsar-bookkeeper-verify-clusterid container :
```
JMX enabled by default │
│ Error: Could not find or load main class "
```
**To Reproduce**
Steps to reproduce the behavior:
Deploy a minimal setup using the project Helm chart using this file:
```
affinity:
anti_affinity: false
components:
functions: false
proxy: false
toolset: false
pulsar_manager: false
namespace: default
monitoring:
prometheus: false
grafana: false
node_exporter: false
alert_manager: false
bookkeeper:
replicaCount: 2
broker:
replicaCount: 2
```
And this command:
```
helm install pulsar -f pulsar/overrides.yaml --set namespace=default ./pulsar/
```
This will bring up a cluster running 2.5.0.
Then try to upgrade just the bookies to a 2.6.0 candidate image by adding this to the override file:
```
images:
bookie:
repository: kafkaesqueio/pulsar-all-v2.6.0-candidate-1
tag: latest
pullPolicy: IfNotPresent
```
And running this command:
```
helm upgrade pulsar -f pulsar/overrides.yaml --set namespace=default ./pulsar/
```
The the bookie will not start because the init container never completes successfully.
**Expected behavior**
The upgrade should succeed.
**Additional context**
This is caused by https://github.com/apache/pulsar/pull/6579. With this change, the environment variable values are no longer automatically applied to `bkenv.sh` when calling `apply-config-from-env.py`. However, it looks like the `bookkeeper shell whatisinstanceid` depends on these variables to be set.
This pattern of calling the bookeeper shell in the init container is used throughout the Helm chart, so this will actually fail most of the components of the chart. I just noticed it first on bookies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Lanayx edited a comment on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
Lanayx edited a comment on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-647187778
Just got this issue with clean 2.6.0 installation, but only in pulsar-recovery pod (inside init container)
```
020-06-21T21:57:17.304727748Z [conf/bookkeeper.conf] Applying config httpServerEnabled = true
2020-06-21T21:57:17.30478306Z [conf/bookkeeper.conf] Applying config httpServerPort = 8000
2020-06-21T21:57:17.304789783Z [conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
2020-06-21T21:57:17.30479629Z [conf/bookkeeper.conf] Applying config useHostNameAsBookieID = true
2020-06-21T21:57:17.304801054Z [conf/bookkeeper.conf] Applying config zkLedgersRootPath = /ledgers
2020-06-21T21:57:17.304805717Z [conf/bookkeeper.conf] Applying config zkServers = pu-pulsar-zookeeper:2181
2020-06-21T21:57:17.454698773Z JMX enabled by default
2020-06-21T21:57:19.346682488Z Error: Could not find or load main class "
2020-06-21T21:57:22.383129167Z JMX enabled by default
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
sijie closed issue #7243:
URL: https://github.com/apache/pulsar/issues/7243
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rvashishth edited a comment on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
rvashishth edited a comment on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-649518996
@sijie @cdbartholomew it seems a similar issue was raised earlier https://github.com/apache/pulsar/issues/6355 and was fixed by changing - [changing `PULSAR_MEM` to BOOKIE_MEM for bookie and auto-recovery configmap?](https://github.com/apache/pulsar/issues/6355#issuecomment-587756717). But in the current master bookkeeper is again using `PULSAR_MEM`... could this be the reason we are again seeing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rvashishth commented on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-649518996
@sijie @cdbartholomew do you have a fix or workaround for this? This error comes are very early stage when installing the chart, would be good to know a fix about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-650269146
@Lanayx @rvashishth: Please use the latest master of https://github.com/apache/pulsar-helm-chart which includes the change of apache/pulsar-helm-chart#26. We will release a helm chart and publish it to https://pulsar.apache.org/charts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Lanayx commented on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
Lanayx commented on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-647187778
Just got this issue with clean 2.6.0 installation, but only on pulsar-recovery pod
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cdbartholomew commented on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
cdbartholomew commented on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-643460326
Sure, the Helm chart can be updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rvashishth edited a comment on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
rvashishth edited a comment on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-649518996
@sijie @cdbartholomew it seems a similar issue was raised earlier https://github.com/apache/pulsar/issues/6355 and was fixed by changing - [changing `PULSAR_MEM` to `BOOKIE_MEM` for bookie and auto-recovery configmap?](https://github.com/apache/pulsar/issues/6355#issuecomment-587756717). But in the current master bookkeeper is again using `PULSAR_MEM`... could this be the reason we are again seeing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7243: Bookies failing to start with Helm chart
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7243:
URL: https://github.com/apache/pulsar/issues/7243#issuecomment-643017288
> With this change, the environment variable values are no longer automatically applied to bkenv.sh when calling apply-config-from-env.py.
`apply-config-from-env.py` is taking environment variables and applying them to the `bkenv.sh`. Since these variables are already environment variables, they can be imported when the script sources the `bkenv.sh` file. We should just use the bash scripts to take the system environments instead of using our scripts to apply those changes.
If there is a problem with the helm chart, we should update the helm chart.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org