[jira] [Commented] (SAMZA-1786) Introduce metadata store abstraction.

["Shanthoosh Venkataraman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558916#comment-16558916 ] 

Shanthoosh Venkataraman commented on SAMZA-1786:
------------------------------------------------

Documenting the tests done for validating the patch changes in yarn:

1. Added unit tests for new classes that were added(Fixed the existing unit tests in LocalityManager/TaskAssignmentManager).
2. All these changes are validated with test jobs present in samza-hello-samza(https://github.com/apache/samza-hello-samza).
3. LinkedIn production job(maes-tests-host-affinity) is verified with these changes to validate if things work end-to-end properly.

> Introduce metadata store abstraction.
> -------------------------------------
>
>                 Key: SAMZA-1786
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1786
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Major
>
> As a part of SEP-11, this JIRA introduces MetadataStore interface to store task and container locality for both yarn and standalone deployment models. Refer to SEP-11 design doc for more details. 
>  
> Few important points to note:
>  # As a part of this changes, LocalityManager/TaskAsssignemtnManager alone will be updated to use this interface(subsequently in upcoming future RB's other util classes will be moved to use this interface as well).
>  # In an immediate followup RB, ZkMetadataStore(storing metadata information in zookeeper) will be added. It will be used in standalone to read/write locality(through LocalityManager & other standard util classes).
>  # In future, executionPlan, other job related metadata(with respect to running containers, stream graph) can be stored in any store through the same abstraction. In future, if we want to deploy standalone jobs in azure, only we need to implement this interface(CosmosDBMetadataStore to store samza metadata in cosmos).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)