You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/07 10:15:31 UTC
[GitHub] [pulsar] JackrayWang opened a new issue #14578: add new node for broker error
JackrayWang opened a new issue #14578:
URL: https://github.com/apache/pulsar/issues/14578
zk
```
[zk: localhost:2181(CONNECTED) 1] ls /
[admin, bookies, counters, ledgers, loadbalance, managed-ledgers, namespace, pulsar, schemas, stream, zookeeper]
[zk: localhost:2181(CONNECTED) 2]
```
bookkeeper
`/opt/pulsar/bin/bookkeeper shell bookiesanity`
` org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Bookie sanity test succeeded
`
But the broker start error. Is some config error? The error log is as follows.
```
org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send thread for session 0x0.
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 10014ms for session id 0x0
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
2022-03-07T18:02:02,106+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
2022-03-07T18:02:02,106+0800 [main] ERROR org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Failed to create zookeeper client to 192.168.228.22:2181,192.168.228.108:2181,192.168.228.107:2181
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:207) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
2022-03-07T18:02:02,107+0800 [main] ERROR org.apache.bookkeeper.client.BookKeeper - Encountered metadata exceptions on initializing metadata client driver
org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 192.168.228.22:2181,192.168.228.108:2181,192.168.228.107:2181
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:227) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:207) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
... 10 more
2022-03-07T18:02:02,107+0800 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: Failed to initialize metadata client driver
java.io.IOException: Failed to initialize metadata client driver
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:464) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 192.168.228.22:2181,192.168.228.108:2181,192.168.228.107:2181
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:227) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
... 8 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:207) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
... 8 more
2022-03-07T18:02:02,108+0800 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.io.IOException: Failed to initialize metadata client driver
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:782) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: java.io.IOException: Failed to initialize metadata client driver
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:464) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
... 2 more
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 192.168.228.22:2181,192.168.228.108:2181,192.168.228.107:2181
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:227) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
... 2 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:207) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:636) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
... 2 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1067570575
It should be related to some network policies, not the pulsar side. I will close this issue first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #14578:
URL: https://github.com/apache/pulsar/issues/14578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] JackrayWang commented on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
JackrayWang commented on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1061335952
This node could connect to zk，the log is as follows。
zk
```
[zk: localhost:2181(CONNECTED) 1] ls /
[admin, bookies, counters, ledgers, loadbalance, managed-ledgers, namespace, pulsar, schemas, stream, zookeeper]
[zk: localhost:2181(CONNECTED) 2]
```
but broker can not。。this is my doubt。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] JackrayWang commented on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
JackrayWang commented on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1062518093
I have deploy a new cluster..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1060772422
@JackrayWang Looks like your broker is not able to connect to the zookeeper server. You can try to use the zookeeper shell to connect to the zookeeper cluster from the broker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] JackrayWang removed a comment on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
JackrayWang removed a comment on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1062518093
I have deploy a new cluster..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] JackrayWang commented on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
JackrayWang commented on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1061340875
bookkeeper and zk is normal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] JackrayWang edited a comment on issue #14578: add new node for broker error
Posted by GitBox <gi...@apache.org>.
JackrayWang edited a comment on issue #14578:
URL: https://github.com/apache/pulsar/issues/14578#issuecomment-1061335952
This node could connect to zk，the log is as follows。
zk
```
[zk: localhost:2181(CONNECTED) 1] ls /
[admin, bookies, counters, ledgers, loadbalance, managed-ledgers, namespace, pulsar, schemas, stream, zookeeper]
[zk: localhost:2181(CONNECTED) 2]
```
but broker can not。。this is my doubt。@codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org