You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/21 14:21:12 UTC
[GitHub] [incubator-pinot] timebertt opened a new issue #6704: Helm chart: minion in CrashLoopBackOff
timebertt opened a new issue #6704:
URL: https://github.com/apache/incubator-pinot/issues/6704
When installing pinot using the helm chart, the minion immediately goes into `CrashLoopBackOff`.
Error logs from the minion instance:
```text
$ k logs pinot-minion-0
ERROR StatusLogger File not found in file system or classpath: /opt/pinot/conf/pinot-minion-log4j2.xml
ERROR StatusLogger Reconfiguration failed: No configuration found for '7852e922' at 'null' in 'null'
14:14:01.433 [Start a Pinot [MINION]] ERROR org.apache.pinot.tools.admin.command.StartServiceManagerCommand - Failed to start a Pinot [MINION] at 0.241 since launch
java.io.IOException: Unable to delete directory /var/pinot/minion/data.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.minion.MinionStarter.start(MinionStarter.java:146) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.service.PinotServiceManager.startMinion(PinotServiceManager.java:162) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
```
Might be related to https://github.com/apache/incubator-pinot/pull/6430/files#diff-f2f45beb0be7ab7259835411627d8aed40dff1b4a92e53ab186cd07e1a57d7feR146-R148
cc @fx19880617
## Steps to reproduce
1. Add helm repo:
```bash
helm repo add pinot https://raw.githubusercontent.com/apache/incubator-pinot/master/kubernetes/helm
```
2. Install pinot chart using the following command and values:
```bash
helm install pinot -f values.yaml pinot/pinot --version 0.2.3
```
```yaml
image:
repository: apachepinot/pinot
tag: 0.6.0
cluster:
name: pinot
```
3. Check on the cluster's state:
```text
$ k get po
NAME READY STATUS RESTARTS AGE
pinot-broker-0 1/1 Running 1 2m15s
pinot-controller-0 1/1 Running 1 2m14s
pinot-minion-0 0/1 CrashLoopBackOff 4 2m14s
pinot-server-0 1/1 Running 0 2m14s
pinot-zookeeper-0 1/1 Running 0 2m15s
```
4. Observe `pinot-minion-0` fail with the following error log from above.
## Workaround
Setting the chart value `minion.dataDir` to some subdirectory of `minion.persistence.mountPath` fixes the problem:
```yaml
minion:
dataDir: /var/pinot/minion/data/data
```
```
$ k get po
NAME READY STATUS RESTARTS AGE
pinot-broker-0 1/1 Running 1 6m43s
pinot-controller-0 1/1 Running 1 6m42s
pinot-minion-0 1/1 Running 0 114s
pinot-server-0 1/1 Running 0 6m42s
pinot-zookeeper-0 1/1 Running 0 6m43s
```
## Additional information
My cluster is running k8s `v1.20.2` on GCP using CSI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6704: Helm chart: minion in CrashLoopBackOff
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6704:
URL: https://github.com/apache/incubator-pinot/issues/6704#issuecomment-803634950
Can you try with latest pinot tag for minion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] timebertt closed issue #6704: Helm chart: minion in CrashLoopBackOff
Posted by GitBox <gi...@apache.org>.
timebertt closed issue #6704:
URL: https://github.com/apache/incubator-pinot/issues/6704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] timebertt commented on issue #6704: Helm chart: minion in CrashLoopBackOff
Posted by GitBox <gi...@apache.org>.
timebertt commented on issue #6704:
URL: https://github.com/apache/incubator-pinot/issues/6704#issuecomment-803659267
Thanks, that works!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org