[GitHub] zhaijack opened a new pull request #1428: Issue #1117: handle race in concurrent bundle split

[GitBox <gi...@apache.org>]

zhaijack opened a new pull request #1428: Issue #1117: handle race in concurrent bundle split
URL: https://github.com/apache/incubator-pulsar/pull/1428
 
 
   ### Motivation
   
   in `NamespaceService.splitAndOwnBundle()`. 
   The concurrent update of Policies of same namespace will cause some override to bundles value that kept in Zookeeper. And this involves at least 2 rounds of split. 
   e.g.
   1. At beginning, we have 2 bundles, with partitions:[0x00000000, 0x7fffffff, 0xffffffff],
   2. First round of split: We will split the 2 bundles at the same time. In first bundle split, the `NamespaceBundles` value that returned by `NamespaceBundleFactory.splitBundles()` would be: 
   [0x00000000, *0x3fffffff*, 0x7fffffff, 0xffffffff]; 
   while at the same time, the `NamespaceBundles` value for second bundle split would be:
    [0x00000000, 0x7fffffff, *0xbfffffff*, 0xffffffff]
   3. Then the 2 split operations, whoever success `updateNamespaceBundles` last, will override the former one. 
   Here if second split finally success,  it will override the former success value [0x00000000, *0x3fffffff*, 0x7fffffff, 0xffffffff], and the final bundles partitions would be:  [0x00000000, 0x7fffffff, *0xbfffffff*, 0xffffffff];  But we thought it should be [0x00000000, *0x3fffffff*, 0x7fffffff, *0xbfffffff*,  0xffffffff]; 
   4. Second round split:  e.g. try to split to range [0x00000000, 0x3fffffff], it will not find upper boundary 0x3fffffff in actual [0x00000000, 0x7fffffff, 0xbfffffff, 0xffffffff].
   And  this will cause the error:

   
   ```

   java.lang.IllegalArgumentException: Invalid upper boundary for bundle
       at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
   
```
   
   ### Modifications
   
   - Add version in NamespaceBundles,  when update the zookeeper z-node, verify the version, and do read-modify-write retry when there are simultaneous write to the z-node.
   - Add test for it.
   
   ### Result
   
   No user behavior will change. This concurrent bug fixed. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services