[jira] [Resolved] (GOBBLIN-458) Refactor flowConfig resource handler to avoid single restli request handled partially on one machine and then forward to another machine.

["Hung Tran (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hung Tran resolved GOBBLIN-458.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 0.13.0

Issue resolved by pull request #2329
[https://github.com/apache/incubator-gobblin/pull/2329]

>  Refactor flowConfig resource handler to avoid single restli request handled partially on one machine and then forward to another machine. 
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: GOBBLIN-458
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-458
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Kuai Yu
>            Assignee: Kuai Yu
>            Priority: Major
>             Fix For: 0.13.0
>
>
> This change is to consolidate the Restli request handling.
> Because today what would happened after user submit a restli request is below:
>  # Standby node receives the request (flowConfig)
>  # Standby convert flowConfig to flowSpec and saved in flowCatalog
>  # In step 2, after persisting to flowCatalog, the scheduler was notified to handle the real scheduling
>  # Because this is Standby node, we cannot schedule it, so forward the message to the real active node.
>  
> Because the actual restli request handling part now was split into two parts and handled by two different nodes. It implies that we have to add some pingpong messages to ack which steps has been executed, and also it adds the complexity for us to add rollback or partial recover mechanism in the future. 
> Another concern is that in the near future, we might have to refactor Helix load balancing logic based on some partitioning logic. When this happened, we just need to rely on Helix to do load balancing work. At that time today's load balancing mechanism (splitting persistence and scheduling) seems not very practical and could be error prone due to we have to concern about network issue between two nodes while we perform the actual handling logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)