[GitHub] [bookkeeper] horizonzy opened a new issue, #3722: RackAware placement policy can not ensure new ensemble selection succeed when one rack goes down. #23

[GitBox <gi...@apache.org>]

horizonzy opened a new issue, #3722:
URL: https://github.com/apache/bookkeeper/issues/3722

   BUG REPORT
   
   Describe the bug
   
   r1 -> bk1, bk2
   r2 -> bk3
   r3 -> bk4
   enable EnforceMinNumRacksPerWriteQuorum and set minNumRacksPerWriteQuorumConfValue=2
   
   When bk3 or bk4 is quarantined, the new ensemble selection should succeed, because it fulfills min rack is 2.
   However, the new ensemble selection result some times will fail.
   
   Run the following unit test in TestRackawareEnsemblePlacementPolicy.java can reproduce the bug.
   
   final int minNumRacksPerWriteQuorumConfValue = 2;
   conf.setMinNumRacksPerWriteQuorum(minNumRacksPerWriteQuorumConfValue);
   conf.setEnforceMinNumRacksPerWriteQuorum(true);
   
   @Test
       public void testNewEnsemblePolicyWithMultipleRacksV2() throws Exception {
           BookieSocketAddress addr1 = new BookieSocketAddress("127.0.0.1", 3181);
           BookieSocketAddress addr2 = new BookieSocketAddress("127.0.0.2", 3181);
           BookieSocketAddress addr3 = new BookieSocketAddress("127.0.0.3", 3181);
           BookieSocketAddress addr4 = new BookieSocketAddress("127.0.0.4", 3181);
   
           // update dns mapping
           StaticDNSResolver.addNodeToRack(addr1.getHostName(), "/default-region/r1");
           StaticDNSResolver.addNodeToRack(addr2.getHostName(), "/default-region/r1");
           StaticDNSResolver.addNodeToRack(addr3.getHostName(), "/default-region/r2");
           StaticDNSResolver.addNodeToRack(addr4.getHostName(), "/default-region/r3");
           // Update cluster
           Set<BookieId> addrs = new HashSet<BookieId>();
           addrs.add(addr1.toBookieId());
           addrs.add(addr2.toBookieId());
           addrs.add(addr3.toBookieId());
           addrs.add(addr4.toBookieId());
           repp.onClusterChanged(addrs, new HashSet<BookieId>());
   
           try {
               int ensembleSize = 3;
               int writeQuorumSize = 3;
               int ackQuorumSize = 2;
   
               Set<BookieId> excludeBookies = new HashSet<>();
               excludeBookies.add(addr4.toBookieId());
               //excludeBookies.add(addr3.toBookieId());
   
               for (int i = 0; i < 50; ++i) {
                   EnsemblePlacementPolicy.PlacementResult<List<BookieId>> ensembleResponse =
                       repp.newEnsemble(ensembleSize, writeQuorumSize,
                           ackQuorumSize, null, excludeBookies);
                   List<BookieId> ensemble = ensembleResponse.getResult();
               }
           } catch (Exception e ) {
               fail("Can not new ensemble selection succeed");
           }
       }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] horizonzy commented on issue #3722: RackAware placement policy can not ensure new ensemble selection succeed when one rack goes down.

[GitBox <gi...@apache.org>]

horizonzy commented on issue #3722:
URL: https://github.com/apache/bookkeeper/issues/3722#issuecomment-1369859251

   In `selectFromNetworkLocation,` it will randomly pick one bookieNode, exclude the picked bookieNode network location, and continue to pick the next bookieNode randomly. If there are no more bookieNode, throw a BKNotEnoughBookiesException.
   
   ## Example:
   There are three bookie nodes. (bk1,bk2,bk3)
   ### The network location:
   ```
   r1 -> bk1, bk2
   r2 -> bk3
   ```
   enable `EnforceMinNumRacksPerWriteQuorum` and set `minNumRacksPerWriteQuorumConfValue=2`
   
   ### Case1: 
   pick bk1 -> exclude bk1 network location `r1` -> pick bk3 -> exclude bk3 network location `r2` -> pick bk2.
   The ensemble: [bk1, bk3, bk1]
   
   ### Case2:
   pick bk3 -> exclude bk3 network location `r2` -> pick bk1/bk2 -> exclude bk1/bk2 network location `r1` -> the left bookie node is bk2/bk1, it's network location is `r1` -> no more available bookies -> throw an exception.
   
   ### Solution
   We still use the original method to pick bookie nodes; it's random, and the random help to load balance.
   
   When it meets the case2, we will make the bookie nodes list in order by network location, then pick the node.
   The ordering rule: the higher the frequency of the network location, it will on the list head.
   
   Example:
   ```
   r1 -> bk1, bk2
   r2 -> bk3
   ```
   The network location `r1` has two bookie nodes. We will make it at the list head. The network location `r2` is at the list tail.
   When we pick nodes, the bookie nodes list is [bk1, bk2, bk3], it always picks bk1 first, then pick bk3, and picks bk2 finally.
   
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] hangc0276 closed issue #3722: RackAware placement policy can not ensure new ensemble selection succeed when one rack goes down.

["hangc0276 (via GitHub)" <gi...@apache.org>]

hangc0276 closed issue #3722: RackAware placement policy can not ensure new ensemble selection succeed when one rack goes down. 
URL: https://github.com/apache/bookkeeper/issues/3722


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org