[GitHub] samza pull request #892: Samza-2018: Using Rocksdb bulk load only for clean-...

[rmatharu <gi...@git.apache.org>]

GitHub user rmatharu opened a pull request:

    https://github.com/apache/samza/pull/892

    Samza-2018: Using Rocksdb bulk load only for clean-slate restores

    Rocksdb has a problem that its bulk-load option cannot be used on existing stores seamlessly. 
    See https://github.com/facebook/rocksdb/issues/2734
    
    Because of this when host-affinity puts a container onto an old host, the restore path's bulk-load mode fails. 
    
    To fix this, 
    We use rocksdb's bulk load option only when the underlying storeDirectory does not exist. 
    If the storeDirectory exists, then we open the store without bulk-load options. 
    
    Note that, an existing storeDirectory may also be "stale", in which case we delete it, and should ideally be using the bulk load option. We will add this additional optimization separately. 
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rmatharu/samza rocksdbbugfix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #892
    
----
commit 2c09b081d17848a357c0a3a8ecafbe36da6abd39
Author: Ray Matharu <rm...@...>
Date:   2019-01-24T02:16:05Z

    Rocksdb bug fix

----


---

[GitHub] samza pull request #892: Samza-2018: Using Rocksdb bulk load only for clean-...

[asfgit <gi...@git.apache.org>]

Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/892


---