You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/14 08:22:55 UTC
[GitHub] [pulsar] Radiancebobo opened a new issue #14675: pulsar initialize-cluster-metadata failed
Radiancebobo opened a new issue #14675:
URL: https://github.com/apache/pulsar/issues/14675
**Describe the bug**
pulsar initialize-cluster-metadata failed.
I have three machines, hadoop102, hadoop103, hadoop104.
I'm sure zookeeper is working fine.
**To Reproduce**
Steps to reproduce the behavior:
1. Prepare 3 machines with zookeeper
2. Download the package apache-pulsar-2.9.1-bin.tar.gz on https://pulsar.apache.org/en/download/
3. tar -zxvf apache-pulsar-2.9.1-bin.tar.gz -C /opt/
4.vim conf/bookkeeper.conf ,Modify the following configuration：
advertisedAddress、journalDirectory、ledgerDirectories、zkServers。
vim conf/broker.conf，Modify the following configuration：
clusterName、zookeeperServers、configurationStoreServers、advertisedAddress
5.Start the zookeeper instance
6. run:
```
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster \
--zookeeper hadoop102:2181,hadoop103:2181,hadoop104:2181 \
--configuration-store hadoop102:2181,hadoop103:2181,hadoop104:2181 \
--web-service-url http://hadoop102:8080,hadoop103:8080,hadoop104:8080 \
--web-service-url-tls https://hadoop102:8443,hadoop103:8443,hadoop104:8443 \
--broker-service-url pulsar://hadoop102:6650,hadoop103:6650,hadoop104:6650 \
--broker-service-url-tls pulsar+ssl://hadoop102::6651,hadoop103:6651,hadoop104:6651
```
9.see the error:
```
2022-03-14T16:12:38,047+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hadoop102:2181,hadoop103:2181,hadoop104:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7d61eb55
2022-03-14T16:12:38,086+0800 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-03-14T16:12:38,120+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2022-03-14T16:12:38,144+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2022-03-14T16:12:38,166+0800 [main-SendThread(hadoop103:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hadoop103/192.168.18.103:2181.
2022-03-14T16:12:38,167+0800 [main-SendThread(hadoop103:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-03-14T16:12:38,179+0800 [main-SendThread(hadoop103:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.18.102:45984, server: hadoop103/192.168.18.103:2181
2022-03-14T16:12:38,228+0800 [main-SendThread(hadoop103:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hadoop103/192.168.18.103:2181, session id = 0x300016b5f340000, negotiated timeout = 30000
2022-03-14T16:12:38,236+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
Exception in thread "main" org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:89)
at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.newInstance(MetadataStoreFactoryImpl.java:52)
at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.createExtended(MetadataStoreFactoryImpl.java:36)
at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:43)
at org.apache.pulsar.PulsarClusterMetadataSetup.initMetadataStore(PulsarClusterMetadataSetup.java:338)
at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:203)
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.addWatch(ZooKeeper.java:3192)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.access$3301(PulsarZooKeeperClient.java:74)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1130)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1124)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$ZooWorker.syncCallWithRetries(PulsarZooKeeperClient.java:1529)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.addWatch(PulsarZooKeeperClient.java:1124)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:82)
... 5 more
2022-03-14T16:12:38,274+0800 [main-SendThread(hadoop103:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x300016b5f340000 for sever hadoop103/192.168.18.103:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x300016b5f340000, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
```
**Expected behavior**
pulsar initialize-cluster-metadata success
**Screenshots**
Configuration on hadoop102：
```
cat conf/bookkeeper.conf | grep -E "advertised|journalDir|ledgerDirectories|zkServers" | grep -v "#"
journalDirectory=/opt/module/pulsar-2.9.1/data/journal
advertisedAddress=hadoop102
ledgerDirectories=/opt/module/pulsar-2.9.1/data/ledger
zkServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
cat conf/broker.conf | grep -E "clusterName|zookeeperServers|configurationStoreServers|advertisedAddress" | grep -v "#"
zookeeperServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
configurationStoreServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
advertisedAddress=hadoop102
clusterName=pulsar-cluster
```
Configuration on hadoop103：
```
cat conf/bookkeeper.conf | grep -E "advertised|journalDir|ledgerDirectories|zkServers" | grep -v "#"
journalDirectory=/opt/module/pulsar-2.9.1/data/journal
advertisedAddress=hadoop103
ledgerDirectories=/opt/module/pulsar-2.9.1/data/ledger
zkServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
cat conf/broker.conf | grep -E "clusterName|zookeeperServers|configurationStoreServers|advertisedAddress" | grep -v "#"
zookeeperServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
configurationStoreServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
advertisedAddress=hadoop103
clusterName=pulsar-cluster
```
Configuration on hadoop104：
```
cat conf/bookkeeper.conf | grep -E "advertised|journalDir|ledgerDirectories|zkServers" | grep -v "#"
journalDirectory=/opt/module/pulsar-2.9.1/data/journal
advertisedAddress=hadoop104
ledgerDirectories=/opt/module/pulsar-2.9.1/data/ledger
zkServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
cat conf/broker.conf | grep -E "clusterName|zookeeperServers|configurationStoreServers|advertisedAddress" | grep -v "#"
zookeeperServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
configurationStoreServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
advertisedAddress=hadoop104
clusterName=pulsar-cluster
```
**Additional context**
I have used apache-pulsar-2.8.1-bin.tar.gz and did not find this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Radiancebobo commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
Radiancebobo commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066605556
> Did you use the built-in ZK of Pulsar?
No, is external zk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Radiancebobo closed issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
Radiancebobo closed issue #14675:
URL: https://github.com/apache/pulsar/issues/14675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower edited a comment on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
BewareMyPower edited a comment on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066959880
I see your ZK client version is 3.5.7, maybe your ZK server is also the same version. The error is usually caused by the wrong combination of Curator in combination with Zookeeper.
Reference: https://stackoverflow.com/questions/35734590/apache-curator-unimplemented-errors-when-trying-to-create-znodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066959880
I see your ZK client version is 3.5.7. The error is usually caused by the wrong combination of Curator in combination with Zookeeper.
Reference: https://stackoverflow.com/questions/35734590/apache-curator-unimplemented-errors-when-trying-to-create-znodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066551341
Did you use the built-in ZK of Pulsar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066523135
The `initialize-cluster-metadata` command only operates on the ZK cluster, so it's not affected by `bookkeeper.conf` or `broker.conf`.
I've ran the following command on a single node ZK cluster in my local env and it works well.
```bash
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster \
--zookeeper localhost:2181 \
--configuration-store localhost:2181 \
--web-service-url http://hadoop102:8080,hadoop103:8080,hadoop104:8080 \
--web-service-url-tls https://hadoop102:8443,hadoop103:8443,hadoop104:8443 \
--broker-service-url pulsar://hadoop102:6650,hadoop103:6650,hadoop104:6650 \
--broker-service-url-tls pulsar+ssl://hadoop102::6651,hadoop103:6651,hadoop104:6651
```
Could you try to specific a single node for `--zookeeper` and `--configuration-store` options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Radiancebobo commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
Radiancebobo commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1068534376
Confirm this problem is caused by version inconsistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Radiancebobo commented on issue #14675: pulsar initialize-cluster-metadata failed
Posted by GitBox <gi...@apache.org>.
Radiancebobo commented on issue #14675:
URL: https://github.com/apache/pulsar/issues/14675#issuecomment-1066542864
I've tried it, but it still doesn't work.
I'm sure zookeeper is ok：
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org