[jira] [Commented] (STORM-876) Dist Cache: Basic Functionality

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986681#comment-14986681 ] 

ASF GitHub Bot commented on STORM-876:
--------------------------------------

GitHub user redsanket opened a pull request:

    https://github.com/apache/storm/pull/845

    [STORM-876] Blobstore API

    Please take a look at the blobstore API. The design docs have been uploaded earlier under the same JIRA. Currently, the blobstore API also works with Nimbus-HA. Apart from the unit tests written for blobstore and nimbus leader election, I have performed few manual tests. Having the PR early looking for suggestions and any missed failover scenarios while the work continues on other additional integration tests and better interface. The manual tests that have been performed are
    1. CRUD operations for the blobstore API. Tested for the Local and Hdfs blobstore operations to work with Nimbus-HA and whether the other nimbodes mirror the operations performed on master nimbus.
    2. Brought up 3 nimbodes and launched a word count topology. Killed the leader daemon and checked whether other non-leader takes up the leadership and continues to run.
    3. Tested whether topology waits for replication to take place before it is launched. For hdfs blobstore the blobstore.replication should be set greater than or equal to topology.min.replication to work effectively.
    4. The blobstore with nimbus-ha does not guarantee updates to sync effectively in the case of failover.
    
    Follow up code to be written.
    1. Integration test for blobstore with nimbus-ha
    2. Move the sync thread in nimbus to blobstore. Waiting on pacemaker push back as it uses some of its code and write the callback code for blobstore

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/redsanket/storm blobstore

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/845.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #845
    
----
commit 6e1ee30ed6e142033a8b3f264918806630da1bf1
Author: Sanket <sc...@untilservice-lm>
Date:   2015-11-02T19:59:13Z

    Squashed commit of the following:
    
    commit 3403a6da661f5ff002cf5e4571853f9b95d8e3c8
    Author: Sanket <sc...@untilservice-lm>
    Date:   Mon Nov 2 13:38:14 2015 -0600
    
        resolved unit test errors caused due to merge commits
    
    commit 898c1d64b5dcbcca2d1efc9a78e731316c1292c0
    Merge: 9909a50 8ba776b
    Author: Sanket <sc...@untilservice-lm>
    Date:   Mon Nov 2 11:41:13 2015 -0600
    
        resolved merge conflicts
    
    commit 9909a50a8b1037e16b21f45ee88c8237f33bd7b8
    Author: Sanket <sc...@untilservice-lm>
    Date:   Mon Nov 2 10:57:29 2015 -0600
    
        cleaned-up debug statements
    
    commit 3f2e1796a74799702113303e896d353ce4218c29
    Author: Sanket <sc...@untilservice-lm>
    Date:   Fri Oct 30 23:25:21 2015 -0500
    
        zookeeper state code and code clean up
    
    commit ac4db652aaad8f3ec71aca50eb31579bf90bd887
    Author: Sanket <sc...@untilservice-lm>
    Date:   Thu Oct 29 11:26:17 2015 -0500
    
        made changes to the sync blob code to fix race condition
    
    commit c96ffcaac5d247da1c77a1e1699b2d7b7f76ebf5
    Author: Sanket <sc...@untilservice-lm>
    Date:   Tue Oct 27 13:38:17 2015 -0500
    
        resolved unit test errors
    
    commit f8f7ebab8cb636a296d84b7e719c368e988063c8
    Author: Sanket <sc...@untilservice-lm>
    Date:   Mon Oct 26 22:51:11 2015 -0500
    
        added callback function
    
    commit c5f02cd54ef50039a6f4dde8f073e839b9ef7b3a
    Author: Sanket <sc...@untilservice-lm>
    Date:   Fri Oct 23 16:20:41 2015 -0500
    
        merge commit, modified unit tests to suit blob-store
    
    commit fdc77d54054470fdaa419dbedaa0d69fdf1f63ce
    Merge: eb0ea1e f75fdde
    Author: Sanket <sc...@untilservice-lm>
    Date:   Wed Oct 21 13:28:46 2015 -0500
    
        modified blobstore to accomodate nimbus ha
    
    commit eb0ea1e167ebfee94e3a0911f05c2102cdeb77b8
    Author: Sanket <sc...@untilservice-lm>
    Date:   Wed Oct 21 13:08:22 2015 -0500
    
        integrated blobstore code
    
    commit f25e7ba8139b0e2ed56ab532339d0b15b4563c6e
    Author: Sanket <sc...@untilservice-lm>
    Date:   Thu Oct 15 19:06:52 2015 -0500
    
        initial commit
    
    commit ba74039b93ee34067e2511e280a50e3958dc8741
    Author: Sanket <sc...@untilservice-lm>
    Date:   Thu Oct 8 15:04:42 2015 -0500
    
        modified thrift files to include blobstore related classes
    
    commit 73859f70e21e4acde0b3b906b53786c7d2b7a5c5
    Author: Sanket <sc...@untilservice-lm>
    Date:   Thu Oct 8 15:01:46 2015 -0500
    
        integrating blobstore and making sure it builds

commit 2506665b02d8808efd20760f1a11e99239518496
Author: Sanket <sc...@untilservice-lm>
Date:   2015-11-02T20:34:09Z

    removed log message and fixed a unit test

commit e8093acbecdfc8893bd9ac571ee8e48f3d610e80
Author: Sanket <sc...@untilservice-lm>
Date:   2015-11-03T03:32:33Z

    removed date change generated files

commit 04458b8f8d3ecb022995281630c130da30dd7b28
Author: Sanket <sc...@untilservice-lm>
Date:   2015-11-03T04:13:01Z

    indentation edit

commit a26bdf8efd94d26ba65489491594a712381373c7
Author: Sanket <sc...@untilservice-lm>
Date:   2015-11-03T04:16:06Z

    indentation edit

----


> Dist Cache: Basic Functionality
> -------------------------------
>
>                 Key: STORM-876
>                 URL: https://issues.apache.org/jira/browse/STORM-876
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>         Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and downloading dist cache items.  storm-core.ser, storm-conf.ser and storm.jar should be written into the blob store instead of residing locally. We need a default implementation of the blob store that does essentially what nimbus currently does and does not need anything extra.  But having an HDFS backend too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and provide a working directory for the worker process with symlinks to the blobs.  It should also allow the blobs to be updated and switch the symlink atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process of getting it ready to push back to open source shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)