You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/11 19:54:56 UTC

[GitHub] [pulsar] michaeljmarshall opened a new issue, #18012: PIP-212: Default reppDnsResolverClass to ZkBookieRackAffinityMapping

michaeljmarshall opened a new issue, #18012:
URL: https://github.com/apache/pulsar/issues/18012

   ### Motivation
   
   The current Bookkeeper configuration defaults to using `org.apache.bookkeeper.net.ScriptBasedMapping` for the `DNSToSwitchMapping` implementation. However, this default configuration does not align with the Broker's default configuration, which is `org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping`. As such, the default configuration for a Pulsar cluster does not lead to ideal rack awareness when ledgers need to be recovered. The result is that a user can configure a cluster for rack awareness and the brokers will honor that configuration, but the autorecovery process will not because it does not have the correct bookkeeper cluster topology view.
   
   I propose we configure bookkeeper to use the broker's `ZkBookieRackAffinityMapping` class. That way, autorecovery will honor the operator's configured rack awareness policies out of the box.
   
   ### Goal
   
   Ensure consistent rack awareness policies.
   
   I propose this is a bug fix that requires patching all active versions of Pulsar.
   
   ### API Changes
   
   _No response_
   
   ### Implementation
   
   See https://github.com/apache/pulsar/pull/15640.
   
   Add default value for `reppDnsResolverClass` to the `conf/bookkeeper.conf` configuration. This change effectively switches the default from `org.apache.bookkeeper.net.ScriptBasedMapping` to `org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping`.
   
   ### Alternatives
   
   The tradeoff is that a user relying on the `ScriptBasedMapping` default might accidentally get switched to using the `ZkBookieRackAffinityMapping` implementation. Given that `ScriptBasedMapping` doesn't work out of the box, and that the broker's default to `ZkBookieRackAffinityMapping`, I think this is an acceptable tradeoff.
   
   ### Anything else?
   
   I manually verified that the `ZkBookieRackAffinityMapping` works by running some tests in a minikube cluster deployed with the DataStax helm chart for Apache Pulsar. I set up 3 racks, 4 bookies, and a topic with a E=2, Qw=2, and Qa=2. I then verified that the autorecovery pod correctly discovered the racks and then identified when an ensemble was not following the rack placement policy after two bookies were removed. I documented my testing a bit more here: https://github.com/datastax/pulsar-helm-chart/pull/214.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yangl commented on issue #18012: PIP-212: Default reppDnsResolverClass to ZkBookieRackAffinityMapping

Posted by GitBox <gi...@apache.org>.
yangl commented on issue #18012:
URL: https://github.com/apache/pulsar/issues/18012#issuecomment-1275858006

   `ZkBookieRackAffinityMapping` implement has problem when the bk and the broker use the different zookeeper cluster, would you verify this case ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on issue #18012: PIP-212: Default reppDnsResolverClass to ZkBookieRackAffinityMapping

Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #18012:
URL: https://github.com/apache/pulsar/issues/18012#issuecomment-1276492770

   I think it wouldn't work in that scenario, though I think we could make it work. The main question is where the zk mapping is stored. I believe it is currently stored in the broker's zk cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] michaeljmarshall closed issue #18012: PIP-212: Default reppDnsResolverClass to ZkBookieRackAffinityMapping

Posted by GitBox <gi...@apache.org>.
michaeljmarshall closed issue #18012: PIP-212: Default reppDnsResolverClass to ZkBookieRackAffinityMapping
URL: https://github.com/apache/pulsar/issues/18012


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org