[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15791957#comment-15791957 ] 

stack commented on HBASE-14614:
-------------------------------

Pushed first cut at core assignment made from patches pulled from [~mbertozzi]'s repo (as per his guidance).

Here's the change log:

{code}
    HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
    Below are commits from Matteo's repo adding in core AM.
    Adds running of remote procedure. Adds batching of remote calls.
    Adds support for assign/unassign in procedures. Adds version info
    reporting in rpc. Adds start of an AMv2.

    First is from
    https://github.com/matteobertozzi/hbase/commit/3622cba4e331d2fc7bfc1932abb4c9cbf5802efa

    Also adding in RS version info from
        https://github.com/matteobertozzi/hbase/commit/ddb4df3964e8298c88c0210e83493aa91ac0942d.patch

    And remote dispath https://github.com/matteobertozzi/hbase/commit/186b9e7c4dae61a79509a6c3aad7f80ec61345e5

    A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
      Dispatch remote procedures every 150ms or 32 items -- which ever
      happens first (configurable). Runs a timeout thread.
      Carries notion of a remote procedure and of a buffer full of these.

    M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
      Add execute procedures call.

    M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
      Adds assign and unassign support.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
      Adds getting RS version out of RPC
      Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
      Add start/stop of remote precedure engine. Add reference to AM2.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
      Extract version number of the server making rpc.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
      Add new assign procedure.
      There can only be one RegionTransitionProcedure per region running at the time,
       since each procedure takes a lock on the region.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
      A procedure-based AM (AMv2).
      TODO
       - handle region migration
       - handle meta assignment first
       - handle sys table assignment first (e.g. acl, namespace)
       - handle table priorities

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
      Adds new move region procedure.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
      Store region state (in hbase:meta by default).

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
      In-memory state of all regions.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
      Base RIT procedure.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
      Unassign procedure.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
      Run region assignement in a manner that pays attention to target
      server version.
{code}


> Procedure v2: Core Assignment Manager
> -------------------------------------
>
>                 Key: HBASE-14614
>                 URL: https://issues.apache.org/jira/browse/HBASE-14614
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>    Affects Versions: 2.0.0
>            Reporter: Stephen Yuan Jiang
>            Assignee: Matteo Bertozzi
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14614.master.001.patch
>
>
> New AssignmentManager implemented using proc-v2.
>  - AssignProcedure handle assignment operation
>  - UnassignProcedure handle unassign operation
>  - MoveRegionProcedure handle move/balance operation
> Concurrent Assign operations are batched together and sent to the balancer
> Concurrent Assign and Unassign operation ready to be sent to the RS are batched together
> This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of HBASE-14616



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)