You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "danielgerendas (via GitHub)" <gi...@apache.org> on 2023/08/02 08:42:41 UTC

[GitHub] [pulsar] danielgerendas opened a new issue, #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

danielgerendas opened a new issue, #20918:
URL: https://github.com/apache/pulsar/issues/20918

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Version
   
   Pulsar version: 3.0.0
   
   ### Minimal reproduce step
   
   We are trying to mitigate the following warning message from bookkeeper and attempted to set the RackAware policy on the bookies, but we couldn't make it to work.
   `New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`
   
   Setup running in AWS:
   3 Bookkeeper instance on each availability zone (us-east-1a, us-east-1b, us-east-1c)
   3 Zookeeper (us-east-1a, us-east-1b, us-east-1c)
   2 Broker (us-east-1a, us-east-1b)
   2 Proxy (us-east-1b, us-east-1c)
   
   Bookies:
   ```
   [root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies list-bookies
   {
     "bookies" : [ {
       "bookieId" : "172.16.1.112:3181"
     }, {
       "bookieId" : "172.16.2.190:3181"
     }, {
       "bookieId" : "172.16.3.6:3181"
     } ]
   }
   ```
   
   Set up racks placement with the following command:
   ```
   $PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.2.190:3181 --group test --hostname 172.16.2.190:3181 --rack /us-east-1a
   $PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.1.112:3181 --group test --hostname 172.16.1.112:3181 --rack /us-east-1b
   $PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.3.6:3181 --group test --hostname 172.16.3.6:3181 --rack /us-east-1c
   ```
   
   Get the rack placement for the bookies:
   ```
   [root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies racks-placement
   test   
    {
        172.16.2.190:3181=BookieInfoImpl(rack=/us-east-1a, hostname=172.16.2.190:3181), 
        172.16.1.112:3181=BookieInfoImpl(rack=/us-east-1b, hostname=172.16.1.112:3181), 
        172.16.3.6:3181=BookieInfoImpl(rack=/us-east-1c, hostname=172.16.3.6:3181)
   }
   ```
   
   ### What did you expect to see?
   
   Something in the logs that tells the policy applied on the bookies.
   
   ### What did you see instead?
   
   Nothing new, only the this warning again, like 50 times:
   `New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`
   
   ### Anything else?
   
   Configs:
   [broker_conf.txt](https://github.com/apache/pulsar/files/12238023/broker_conf.txt)
   [bookkeeper_conf.txt](https://github.com/apache/pulsar/files/12238025/bookkeeper_conf.txt)
   
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1670765897

   These logs were extracted from DataDog, the `services` column shows which pulsar service logged out the message (manager, broker, proxy, bookkeeper, zookeeper)
   
   The `reppDnsResolverClass` was already set to `org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping`, so this is not the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Bookkeeper rackaware placement policy not applying on bookies [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-6381739257

   Thanks @danielgerendas for the report and the logs, and @CaveEcho for the extra data point. Both reports are on releases that are no longer supported (3.0.0 and 2.7.5), and the rack-mapping code has changed a lot since then, including https://github.com/apache/pulsar/pull/25067, which fixed a race where the placement policy could resolve a bookie's rack before the rack mapping was updated (#25058) and log this kind of "not adhering to Placement Policy" warning. Could you check whether this still happens on a currently supported release (4.0.x, 4.2.x or 5.0.x)? If it does, please share which process logs the warning (broker or bookie/AutoRecovery), that process's `ensemblePlacementPolicy`, `reppDnsResolverClass` and metadata store settings (`metadataServiceUri`/`zkServers`) next to the broker's `metadataStoreUrl`, and the output of `pulsar-admin bookies racks-placement`. Without an update, this issue will be closed in a few weeks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1668869392

   The log content is too short, I can't find the problems.
   
   According to your context.
   ```
   [root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies racks-placement
   test   
    {
        172.16.2.190:3181=BookieInfoImpl(rack=/us-east-1a, hostname=172.16.2.190:3181), 
        172.16.1.112:3181=BookieInfoImpl(rack=/us-east-1b, hostname=172.16.1.112:3181), 
        172.16.3.6:3181=BookieInfoImpl(rack=/us-east-1c, hostname=172.16.3.6:3181)
   }
   ```
   
   It looks like you config the rack info correctly. 
   
   I'm not sure your problem is related with https://github.com/apache/pulsar/pull/18672, could you upgrade the broker version.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1663490226

   > New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []
   
   
   Where did you see these logs? You have config the rack info correctly. 
   
   > Try to set bookkeeperClientRegionawarePolicyEnabled=true in the broker.conf
   
   No need for this.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Bookkeeper rackaware placement policy not applying on bookies [pulsar]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1763232329

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1669153850

   Everything is on v3.0.0, is this issue present in that version too? As I see your linked problem is only fixed in 2.9.x versions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1669227268

   [extract-2023-08-08T09_10_06.971Z.csv](https://github.com/apache/pulsar/files/12289480/extract-2023-08-08T09_10_06.971Z.csv)
   
   These are the logs from the last 1 hour, restart the brokers at 10:50:54.113
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1663807144

   > In your config. The ensemble size is 2. But the log shows that the new ensemble is [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181], which size is 3. The log does not match your config.
   > 
   > ```
   > managedLedgerDefaultEnsembleSize=2
   > managedLedgerDefaultWriteQuorum=2
   > managedLedgerDefaultAckQuorum=2
   > ```
   
   Tried to update managedLedgerDefaultEnsembleSize to 3, but brokers crashed.
   Should I update both property to 3 or just managedLedgerDefaultEnsembleSize?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1663571311

   In your config. The ensemble size is 2. But the log shows that the new ensemble is [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181], which size is 3.  The log does not match your config.
   
   ```
   managedLedgerDefaultEnsembleSize=2
   managedLedgerDefaultWriteQuorum=2
   managedLedgerDefaultAckQuorum=2
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1670562876

   What's the column named `service`, all the warning logs's service are `pulsar-bookkeeper`.
   
   If I'm not mistaken, this log should be printed by AutoRecovery in BookKeeper. In AutoRecovery, the default DNSToSwitchMapping is DefaultResolver, which cannot detect the rack info set in the Pulsar side. If you want BookKeeper to also be aware of the rack info, you need to configure reppDnsResolverClass=org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.
   
   Please note that org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping is defined in Pulsar's code, and you need to start BookKeeper in the Pulsar side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1712380123

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Bookkeeper rackaware placement policy not applying on bookies [pulsar]

Posted by "CaveEcho (via GitHub)" <gi...@apache.org>.
CaveEcho commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-3425443364

   ```
   New ensemble: [pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: []
   ```
   
   I have the same issue. The pulsar version is 2.7.5, but the cluster is still running normally at present. I'm not sure what impact it will have.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1668212533

   > > Tried to update managedLedgerDefaultEnsembleSize to 3, but brokers crashed.
   > 
   > Could you paste the crash logs?
   > 
   > The logs show that the ensemble size already is 3, why does your config is `2:2:2`. It's so strange. Could you confirm the context you supply is right?
   > 
   > ```
   > New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy
   > ```
   
   Set these all to 3, now the brokers doesn't crash, but still getting the ...is not adhering to Placement Policy message.
   What do you mean under context?
   ```
   managedLedgerDefaultEnsembleSize=3
   managedLedgerDefaultWriteQuorum=3
   managedLedgerDefaultAckQuorum=3
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1668223655

   Could you upload the pulsar broker logs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] danielgerendas commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "danielgerendas (via GitHub)" <gi...@apache.org>.
danielgerendas commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1668351525

   > Could you upload the pulsar broker logs?
   
   [pulsar_logs.csv](https://github.com/apache/pulsar/files/12283797/pulsar_logs.csv)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1669179117

   The v3.0.0 already has the fix, maybe there are another problems about it.
   Could you upload the full logs, the file [pulsar_logs.csv](https://github.com/apache/pulsar/files/12283797/pulsar_logs.csv) is incomplete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1719012583

   > These logs were extracted from DataDog, the `services` column shows which pulsar service logged out the message (manager, broker, proxy, bookkeeper, zookeeper)
   > 
   > The `reppDnsResolverClass` was already set to `org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping`, so this is not the problem.
   
   Sorry for the late reply. Is the ZooKeeper address used in your BookKeeper the same as in Pulsar? The bookie information is stored in the ZooKeeper used by Pulsar.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] horizonzy commented on issue #20918: [Bug] Bookkeeper rackaware placement policy not applying on bookies

Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on issue #20918:
URL: https://github.com/apache/pulsar/issues/20918#issuecomment-1664031126

   > Tried to update managedLedgerDefaultEnsembleSize to 3, but brokers crashed.
   
   Could you paste the crash logs?
   
   
   The logs show that the ensemble size already is 3, why does your config is `2:2:2`.   It's so strange. Could you confirm the context you supply is right?
   
   ```
   New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org