[jira] [Created] (GOBBLIN-458) Refactor flowConfig resource handler to avoid single restli request handled partially on one machine and then forward to another machine.

["Kuai Yu (JIRA)" <ji...@apache.org>]

Kuai Yu created GOBBLIN-458:
-------------------------------

             Summary:  Refactor flowConfig resource handler to avoid single restli request handled partially on one machine and then forward to another machine. 
                 Key: GOBBLIN-458
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-458
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Kuai Yu
            Assignee: Kuai Yu


This change is to consolidate the Restli request handling.

Because today what would happened after user submit a restli request is below:
 # Standby node receives the request (flowConfig)
 # Standby convert flowConfig to flowSpec and saved in flowCatalog
 # In step 2, after persisting to flowCatalog, the scheduler was notified to handle the real scheduling
 # Because this is Standby node, we cannot schedule it, so forward the message to the real active node.

 

Because the actual restli request handling part now was split into two parts and handled by two different nodes. It implies that we have to add some pingpong messages to ack which steps has been executed, and also it adds the complexity for us to add rollback or partial recover mechanism in the future. 

Another concern is that in the near future, we might have to refactor Helix load balancing logic based on some partitioning logic. When this happened, we just need to rely on Helix to do load balancing work. At that time today's load balancing mechanism (splitting persistence and scheduling) seems not very practical and could be error prone due to we have to concern about network issue between two nodes while we perform the actual handling logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)