You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by shanthoosh <gi...@git.apache.org> on 2018/07/26 21:52:35 UTC

[GitHub] samza pull request #583: SAMZA-1786: Introduce metadata store abstraction.

GitHub user shanthoosh opened a pull request:

    https://github.com/apache/samza/pull/583

    SAMZA-1786: Introduce metadata store abstraction.

    As a part of SEP-11, this patch adds MetadataStore interface to store task and container locality in both yarn and standalone deployment models. Please refer to SEP-11 for more details. 
    
    Few important points to note:
    1. As a part of this changes, LocalityManager/TaskAsssignmentManager alone will be updated to use this interface(subsequently in upcoming future RB's other util classes will be moved to use this interface as well).
    2. In an immediate followup RB, ZkMetadataStore(storing metadata information in zookeeper) will be added. It will be used in standalone to read/write locality(through LocalityManager & other standard util classes).
    3. In future, ExecutionPlan, streamGraph and other job related metadata can be stored in any custom store through the same abstraction.
    
    Testing:
    1. Added unit tests for new classes introduced in the patch(Fixed the existing unit tests in LocalityManager/TaskAssignmentManager).
    2. All the changes in the patch were validated with test jobs in samza-hello-samza(https://github.com/apache/samza-hello-samza).
    3. LinkedIn testing job(maes-tests-host-affinity) was verified with these changes to validate if things work end-to-end.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shanthoosh/samza metadata_store_iface

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #583
    
----
commit fad4fc09238dda276661a5a201e398980b05caeb
Author: Shanthoosh Venkataraman <sp...@...>
Date:   2018-07-24T18:37:47Z

    Initial version of MetadataStore interface.

----


---

[GitHub] samza pull request #583: SAMZA-1786: Introduce metadata store abstraction.

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/583


---