You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/09/10 08:14:34 UTC
[GitHub] [pulsar] savearray2 opened a new issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
savearray2 opened a new issue #8027:
URL: https://github.com/apache/pulsar/issues/8027
**Describe the bug**
I've set up a basic pulsar cluster and attempted to set bookie rack information via ```pulsar-admin bookies set-bookie-rack```.
The following is the output from ```racks-placement```:
```
{
"default" : {
"koto-bk-aen-0" : {
"rack" : "aen/0"
},
"koto-bk-aen-1" : {
"rack" : "aen/1"
},
"koto-pl-aen-0" : {
"rack" : "aen/0"
},
"koto-pl-aen-1" : {
"rack" : "aen/1"
},
"koto-bk-aen0" : {
"rack" : "aen/0"
},
"koto-bk-aen1" : {
"rack" : "aen/1"
},
"koto-pl-aen0" : {
"rack" : "aen/0"
},
"koto-pl-aen1" : {
"rack" : "aen/1"
},
"koto-pulsar-aen0" : {
"rack" : "aen/0"
},
"koto-pulsar-aen1" : {
"rack" : "aen/1"
}
}
}
```
During the Pulsar broker client startup, I get the following error (please note it says it can't find ```koto-bk-aen-0``` even though it's clearly specified above):
```
07:41:34.078 [main] ERROR org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping, used default subnet resolver : java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
07:41:34.079 [ForkJoinPool.commonPool-worker-1] WARN com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache - Exception thrown during asynchronous load
java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:103) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) ~[?:1.8.0_252]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$0(ZkBookieRackAffinityMapping.java:85) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.TreeMap.forEach(TreeMap.java:1005) ~[?:1.8.0_252]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.updateRacksWithHost(ZkBookieRackAffinityMapping.java:84) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.access$3(ZkBookieRackAffinityMapping.java:78) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:154) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:1) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$13(ZooKeeperCache.java:362) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598) ~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
Caused by: java.net.UnknownHostException: koto-bk-aen-0
at org.apache.bookkeeper.net.BookieSocketAddress.<init>(BookieSocketAddress.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:87) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
... 11 more
```
The bookies are then added to the default region/default rack, ignoring the settings from ```racks-placement```:
```
07:41:34.139 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/koto-bk-aen-0:3181
07:41:34.139 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:172.28.131.158:0> @ /default-region/default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DNSResolverDecorator.
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/koto-bk-aen-1:3181
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/koto-bk-aen-0:3181
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/koto-bk-aen-1:3181
```
Relevant portions of the ```broker.conf``` file:
```
zookeeperServers=koto-aen-zk0:2181,koto-aen-zk1:2181,koto-aen-zk2:2181
configurationStoreServers=koto-cfg-zk0:2184,koto-cfg-zk1:2184,koto-cfg-zk2:2184,koto-cfg-aen-zk0:2184,koto-cfg-aen-zk1:2184,koto-cfg-aen-zk2:2184
bindAddress=0.0.0.0
advertisedAddress=koto-pulsar-aen0
bookkeeperClientRackawarePolicyEnabled=true
bookkeeperClientRegionawarePolicyEnabled=true
bookkeeperClientReorderReadSequenceEnabled=true
bookkeeperClientIsolationGroups=
bookkeeperClientSecondaryIsolationGroups=
bookkeeperClientMinAvailableBookiesInIsolationGroups=
```
**Expected behavior**
The bookies should be added to regions/racks based on the ```pulsar-admin``` settings.
**Version Info:**
- Using the official docker image: ```apachepulsar/pulsar-all:latest```
- Version: ```07:41:32.588 [main] INFO org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service; version: '2.6.1'```
**Additional context**
Not being able to properly specify region/rack settings really defeats the purpose of Pulsar's scalability. Ensuring that Pulsar writes to bookies in different regions is an important feature. Hopefully someone can help with this...
I'd really, really like to get this working :)
Also, if necessary, I'm willing to clone the project and start digging through the source, if contributions are needed/welcome (and this isn't being taken care of elsewhere).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mkozioro edited a comment on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
mkozioro edited a comment on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690296089
Hi @savearray2
I'm not using bookkeeperClientRegionawarePolicyEnabled, only bookkeeperClientRackawarePolicyEnabled
Everything seems to be working fine for me with the newest pulsar 2.6.1 when setting racks using command:
/pulsar/bin/pulsar-admin --admin-url "${PULSAR_BROKER_URL}" bookies set-bookie-rack --bookie "${HOSTNAME_FQDN}:${BOOKKEEPER_BOOKIE_PORT}" --rack "/${ZONE}"
$ZONE for me is azure kubernetes service availability zone so like westeurope-1, westeurope-2,....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
sijie closed issue #8027:
URL: https://github.com/apache/pulsar/issues/8027
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mkozioro commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
mkozioro commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690296089
Hi @savearray2
I'm not using bookkeeperClientRegionawarePolicyEnabled, only bookkeeperClientRackawarePolicyEnabled
Everything seems to be working fine for me with the newest pulsar 2.6.1 when setting racks using command:
/pulsar/bin/pulsar-admin --admin-url "${PULSAR_BROKER_URL}" bookies set-bookie-rack --bookie "${HOSTNAME_FQDN}:${BOOKKEEPER_BOOKIE_PORT}" --rack "/${ZONE}"
$ZONE for us is azure kubernetes service availability zone so like westeurope-1, westeurope-2,....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] savearray2 commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
savearray2 commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690316104
Hi there, @mkozioro
I've rewritten my configuration:
```
# Enable rack-aware bookie selection policy. BK will chose bookies from different racks when
# forming a new bookie ensemble
bookkeeperClientRackawarePolicyEnabled=true
# Enable region-aware bookie selection policy. BK will chose bookies from
# different regions and racks when forming a new bookie ensemble
# If enabled, the value of bookkeeperClientRackawarePolicyEnabled is ignored
bookkeeperClientRegionawarePolicyEnabled=false
# Enable/disable reordering read sequence on reading entries.
bookkeeperClientReorderReadSequenceEnabled=true
```
Here's the output from ```racks-placement```:
```
# ./pulsar-admin bookies racks-placement
{
"default" : {
"koto-bk-aen-0:3181" : {
"rack" : "/aen"
},
"koto-bk-aen-1:3181" : {
"rack" : "/aen"
},
"koto-bk-aen-0" : {
"rack" : "/aen"
},
"koto-bk-aen-1" : {
"rack" : "/aen"
}
}
}
```
I still get the same error (on the broker):
```
14:00:28.490 [main] ERROR org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping, used default subnet resolver : java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
14:00:28.489 [ForkJoinPool.commonPool-worker-1] WARN com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache - Exception thrown during asynchronous load
java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:103) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) ~[?:1.8.0_252]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$0(ZkBookieRackAffinityMapping.java:85) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.TreeMap.forEach(TreeMap.java:1005) ~[?:1.8.0_252]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.updateRacksWithHost(ZkBookieRackAffinityMapping.java:84) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.access$3(ZkBookieRackAffinityMapping.java:78) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:154) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:1) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$13(ZooKeeperCache.java:362) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598) ~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
Caused by: java.net.UnknownHostException: koto-bk-aen-0
at org.apache.bookkeeper.net.BookieSocketAddress.<init>(BookieSocketAddress.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:87) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
... 11 more
```
Which still results in the following:
```
14:00:28.536 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/koto-bk-aen-0:3181
14:00:28.536 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/koto-bk-aen-1:3181
```
Could I ask you a question?
I'm currently using the following as part of my ```bookkeeper.conf```:
```
# Port that bookie server listen on
bookiePort=3181
# Configure a specific hostname or IP address that the bookie should use to advertise itself to
# clients. If not set, bookie will advertised its own IP address or hostname, depending on the
# listeningInterface and useHostNameAsBookieID settings.
advertisedAddress=koto-bk-aen-0
# Whether the bookie should use its hostname to register with the
# co-ordination service(eg: Zookeeper service).
# When false, bookie will use its ip address for the registration.
# Defaults to false.
useHostNameAsBookieID=true
```
Do you include the port (```:3181```) on the ```advertisedAddress``` value (e.g. ```advertisedAddress=koto-bk-aen-0:3181```)? I'm wondering if this has something to do with the problem...
As an example my bookkeeper node starts up with the following log:
```
13:51:30.989 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=koto-bk-aen-0
```
Also included in the bookkeeper node log is this:
```
13:51:34.855 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/koto-bk-aen-0:3181
13:51:34.855 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/koto-bk-aen-1:3181
```
But I'm not sure if this matters for bookkeeper...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] savearray2 commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
savearray2 commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690376334
For anyone else who ever runs into this problem. I've figured it out. This has been plaguing me for days.
I followed the stack trace to the following line:
https://github.com/apache/bookkeeper/blob/2f08377f5c56f96389fb3a8e51844844537e118b/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/BookieSocketAddress.java#L68
In short, EVERY address added via the ```pulsar-admin bookies set-bookie-rack``` command MUST be in the format of ```<FQDN>:<port>```. If it's not in the correct format (if there's no ```:``` found), it'll throw an ```UnknownHostException```. This should probably be documented much better, but I'm glad to at least get the fix on record.
```java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0``` is not very descriptive.
E.g. (the proper format):
```
{
"default" : {
"koto-bk-aen-0:3181" : {
"rack" : "/aen"
},
"koto-bk-aen-1:3181" : {
"rack" : "/aen"
}
}
}
```
Thanks to @mkozioro for helping. If I wasn't told this was working with someone else's configuration, I never would have followed the stack trace.
--
I'll leave it up to the maintainers of the project on whether or not they want to update the documentation with this information... If not, this issue can be closed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690831275
@savearray2 Yes. We should at least update the help messages of `bookies set-bookie-rack`.
```
In short, EVERY address added via the pulsar-admin bookies set-bookie-rack command MUST be in the format of <FQDN>:<port>. If it's not in the correct format (if there's no : found), it'll throw an UnknownHostException. This should probably be documented much better, but I'm glad to at least get the fix on record.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-690831275
@savearray2 Yes. We should at least update the help messages of `bookies set-bookie-rack`.
```
In short, EVERY address added via the pulsar-admin bookies set-bookie-rack command MUST be in the format of <FQDN>:<port>. If it's not in the correct format (if there's no : found), it'll throw an UnknownHostException. This should probably be documented much better, but I'm glad to at least get the fix on record.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8027: Issue applying region/rack information (ZkBookieRackAffinityMapping)
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8027:
URL: https://github.com/apache/pulsar/issues/8027#issuecomment-708880065
The help message has been updated. Close this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org