You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/24 22:47:03 UTC
[GitHub] [pulsar] oldthreefeng opened a new issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
oldthreefeng opened a new issue #10692:
URL: https://github.com/apache/pulsar/issues/10692
**Describe the bug**
A clear and concise description of what the bug is.
pulsar 2.7.1
```
k get pod -n pulsar
NAME READY STATUS RESTARTS AGE
pulsar-bookie-0 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-1 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-2 0/1 CrashLoopBackOff 73 5h58m
pulsar-bookie-3 0/1 CrashLoopBackOff 73 5h58m
pulsar-broker-0 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-1 0/1 CrashLoopBackOff 71 5h58m
pulsar-broker-2 0/1 CrashLoopBackOff 71 5h57m
pulsar-proxy-0 1/1 Running 0 5h57m
pulsar-proxy-1 1/1 Running 0 5h57m
pulsar-proxy-2 1/1 Running 0 5h57m
pulsar-pulsar-manager-6bf99b4d88-jk22g 1/1 Running 0 27d
pulsar-recovery-0 1/1 Running 1 27d
pulsar-toolset-0 1/1 Running 1 27d
pulsar-zookeeper-0 1/1 Running 1 27d
pulsar-zookeeper-1 1/1 Running 1 27d
pulsar-zookeeper-2 1/1 Running 1 27d
```
bookie
```
k logs -n pulsar pulsar-bookie-0 --tail 20 -f
22:20:20.891 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22:20:20.895 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
22:20:20.904 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
22:20:20.918 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/172.16.35.208:2181. Will not attempt to authenticate using SASL (unknown error)
22:20:20.924 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.16.67.24:45748, server: pulsar-zookeeper/172.16.35.208:2181
22:20:20.931 [main-SendThread(pulsar-zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/172.16.35.208:2181, sessionid = 0x2000001f885027b, negotiated timeout = 30000
22:20:20.934 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
22:20:21.104 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current]
22:20:21.104 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
22:20:21.113 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:476) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:712) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
```
broker
```
22:28:45.687 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal Server Errorjava.io.ByteArrayInputStream@7c3c22f9
```
so is there anyone who can help me out?
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: centos 7.9
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-848346982
The reason that `Invalid cookie` looks like you clean up the data directory but not reinitialize the cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] oldthreefeng closed issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
oldthreefeng closed issue #10692:
URL: https://github.com/apache/pulsar/issues/10692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-1058890060
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] oldthreefeng commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-848420386
@zymap so re run the `pulsar-bookie-init` job can fix the problem?
BTW, in our test env, it Crash in the middle night , no one clean up the data directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap edited a comment on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
zymap edited a comment on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-847428755
You might need to clean up the `pvc` and `pv`. Use `k delete pvc --all -n pulsar` to clean up and then start the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-847428755
You might need to clean up the `pvc` and `pv`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] oldthreefeng commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-847506569
fix by recreate whole cluster
```
k get pod -n pulsar
NAME READY STATUS RESTARTS AGE
pulsar-bookie-0 1/1 Running 0 7m29s
pulsar-bookie-1 1/1 Running 0 7m29s
pulsar-bookie-2 1/1 Running 0 7m29s
pulsar-bookie-3 1/1 Running 0 7m29s
pulsar-bookie-init-9rk8j 0/1 Completed 0 7m29s
pulsar-broker-0 1/1 Running 0 7m29s
pulsar-broker-1 1/1 Running 0 7m29s
pulsar-broker-2 1/1 Running 0 7m29s
pulsar-proxy-0 1/1 Running 0 7m29s
pulsar-proxy-1 1/1 Running 0 7m29s
pulsar-proxy-2 1/1 Running 0 7m29s
pulsar-pulsar-init-ppjc4 0/1 Completed 0 7m29s
pulsar-pulsar-manager-6bf99b4d88-d6jnk 1/1 Running 0 7m30s
pulsar-recovery-0 1/1 Running 0 7m30s
pulsar-toolset-0 1/1 Running 0 7m29s
pulsar-zookeeper-0 1/1 Running 0 7m29s
pulsar-zookeeper-1 1/1 Running 0 7m11s
pulsar-zookeeper-2 1/1 Running 0 6m51s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-848746897
Yes. It should fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] oldthreefeng commented on issue #10692: pulsar-bookie and pulsar-broker CrashLoopBackOff in kubernetes
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #10692:
URL: https://github.com/apache/pulsar/issues/10692#issuecomment-847489173
@zymap but why this happed ? its not a good idea to clean pvc when the pod CrashLoopBackOff.
Its just runing three weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org