[jira] [Created] (HELIX-730) [TASK] Add ThreadCountBasedAssignmentCalculator and integrate with Workflow/JobRebalancer and fix rebalancing logic

["Hunter L (JIRA)" <ji...@apache.org>]

Hunter L created HELIX-730:
------------------------------

             Summary: [TASK] Add ThreadCountBasedAssignmentCalculator and integrate with Workflow/JobRebalancer and fix rebalancing logic
                 Key: HELIX-730
                 URL: https://issues.apache.org/jira/browse/HELIX-730
             Project: Apache Helix
          Issue Type: Improvement
            Reporter: Hunter L


For quota-based scheduling of tasks, we have added the TaskAssigner interface that takes into account AssignableInstances by way of AssignableInstanceManager. In order to use this in the currently-existing pipeline prior to Task Framework 2.0, GenericTaskAssignmentCalculator was replaced with ThreadCountBasedAssignmentCalculator, which is a wrapper around TaskAssigner. Necessary adjustments were made in Workflow/JobRebalancer for this replacement. Also the rebalance logic in Workflow/JobRebalancer was reviewed and fixed. Additionally, TestQuotaBasedScheduling is added to test quota-based task scheduling. Note that quotas will apply to both generic and targeted jobs.

A few bugs were uncovered during this process such as the faulty retry logic that never really got tasks to restart. For more details, see the changelist below:

Changelist:
    1. Add ThreadCountBasedAssignmentCalculator, a wrapper around ThreadCountBasedTaskAssigner
    2. Make logic changes in JobRebalancer to enable the use of ThreadCountBasedAssignmentCalculator
    3. Fix the failing test by using a thread-safe map and rename TestGenericTaskAssignmentCalculator to TestTaskAssignmentCalculator to better reflect what its tests are doing
    4. Add retry logic that was previously absent for INIT and DROPPED tasks in JobRebalancer
    5. Add TestQuotaBasedScheduling to test that jobs and tasks were being assigned and scheduled per quota config set in ClusterConfig
    6. Add more log messages to aid with task-scheduling debugging in AssignableInstance
    7. In AbstractTaskDispatcher, for tasks that are STOPPED, TIMED_OUT, TASK_ERROR, the retry logic was newly implemented so that they get re-started correctly
    8. In AbstractTaskDispatcher, when enforcing overlapAssign for jobs with isAllowOverlapAssignment(), a fix was implemented so that only jobs whose state is IN_PROGRESS are considered
    9. In AbstractTaskDispatcher, isWorkflowFinished() method was modified so that non-active jobs will have their tasks' resource freed from AssignableInstances to prevent resource leak
   10. In markJobFailed() and markJobCompleted(), non-active jobs will have their tasks' resource freed from AssignableInstances to prevent resource leak
   11. Fix the logic so that quotas do not apply to targeted jobs
   12. Fix TestTaskRebalancer (assumes Consistent Hashing, which is no longer used)
   13. Fix TestIndependentTaskRebalancer (assumes Consistent Hashing, no longer used)
   14. Assignment logic was improved so that incomplete tasks whose assigned participants are no longer live will be re-assigned accordingly
   15. Fix TestTaskRebalanceFailover (tasks on non-live instances will be re-assigned promptly)
   16. Fix TestRebalanceRunningTask (targeted jobs will get tasks reassigned upon liveInstance and currentState change)
   17. Fix a bug in FixedAssignmentCalculator and assignment logic for targeted jobs such that a task index will no longer be assigned multiple times
   18. Fix TestJobFailureTaskNotStarted (tasks were not being assigned at all due to having reached maximum capacity for quota)
   19. Add targetedTaskConfigMap field in JobConfig to cache TaskConfig objects for targeted tasks to reduce object creation and GC overload
   20. Fix JobConfig so that it doesn't write quotaType to ZooKeeper when quotaType is null or not set
   21. Fix deleteWorkflow() in TaskUtil so that the earliest delete failure will render the entire method as failed (and return prematurely to prevent breaking other ZNodes from incomplete deletion)
   22. Fix TestDeleteWorkflow by adding another removeProperty() clause to lower failure rate



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)