You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/10/13 17:02:18 UTC

[GitHub] [helix] NealSun96 commented on a change in pull request #1439: Implement addTask API

NealSun96 commented on a change in pull request #1439:
URL: https://github.com/apache/helix/pull/1439#discussion_r504115658



##########
File path: helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
##########
@@ -526,6 +533,185 @@ public void enqueueJobs(final String queue, final List<String> jobs,
     }
   }
 
+  /**
+   * Add task to a running (IN-PROGRESS) job or a job which has not started yet. Timeout for this
+   * operation is the default timeout which is 5 minutes. {@link TaskDriver#DEFAULT_TIMEOUT}
+   * Note1: Task cannot be added if the job is in an illegal state. A job can accept
+   * new task if the job is in-progress or it has not started yet.
+   * Note2: The job can only be added to non-targeted jobs.
+   * Note3: The taskID for the new task should be unique. If not, this API throws an exception.
+   * Note4: In case of timeout exception, it is the user's responsibility to check whether the task
+   * has been successfully added or not.
+   * @param workflowName
+   * @param jobName
+   * @param taskConfig
+   * @throws Exception if there is an issue with the request or the operation. 1-
+   *           IllegalArgumentException will be thrown if the inputs are invalid. 2- HelixException
+   *           will be thrown if the job is not in the states to accept a new task or if there is
+   *           any issue in updating jobConfig. 3- TimeoutException will be thrown if the outcome of
+   *           the task addition is unknown and cannot be verified.
+   */
+  public void addTask(String workflowName, String jobName, TaskConfig taskConfig) throws Exception {
+    addTask(workflowName, jobName, taskConfig, DEFAULT_TIMEOUT);
+  }
+
+  /**
+   * Add task to a running (IN-PROGRESS) job or a job which has not started yet
+   * Note1: Task cannot be added if the job is in an illegal state. A job can accept
+   * new task if the job is in-progress or it has not started yet.
+   * Note2: The job can only be added to non-targeted jobs.
+   * Note3: The taskID for the new task should be unique. If not, this API throws an exception.
+   * Note4: In case of timeout exception, it is the user's responsibility to check whether the task
+   * has been successfully added or not.
+   * Note5: timeout is the time that this API checks whether the task has been successfully added or
+   * not.
+   * @param workflowName
+   * @param jobName
+   * @param taskConfig
+   * @param timeoutMs
+   * @throws Exception if there is an issue with the request or the operation. 1-
+   *           IllegalArgumentException will be thrown if the inputs are invalid. 2- HelixException
+   *           will be thrown if the job is not in the states to accept a new task or if there is
+   *           any issue in updating jobConfig. 3- TimeoutException will be thrown if the outcome of
+   *           the task addition is unknown and cannot be verified.
+   */
+  public void addTask(String workflowName, String jobName, TaskConfig taskConfig, long timeoutMs)
+      throws Exception {
+
+    if (timeoutMs < DEFAULT_SLEEP) {
+      throw new IllegalArgumentException(
+          String.format("Timeout is less than the minimum acceptable timeout value which is %s ms",
+              DEFAULT_SLEEP));
+    }
+
+    long endTime = System.currentTimeMillis() + timeoutMs;
+
+    validateAddTaskConfigs(workflowName, jobName, taskConfig);
+
+    String nameSpaceJobName = TaskUtil.getNamespacedJobName(workflowName, jobName);
+    WorkflowContext workflowContext = getWorkflowContext(workflowName);
+    JobContext jobContext = getJobContext(nameSpaceJobName);
+    if (workflowContext == null || jobContext == null) {
+      // Workflow context or job context is null. It means job has not been started. Hence task can
+      // be added to the job
+      addTaskToJobConfig(workflowName, jobName, taskConfig, endTime);
+      return;
+    }
+
+    TaskState jobState = workflowContext.getJobState(nameSpaceJobName);
+
+    if (jobState == null) {

Review comment:
       This if condition can be combined with the one below. 

##########
File path: helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
##########
@@ -526,6 +533,185 @@ public void enqueueJobs(final String queue, final List<String> jobs,
     }
   }
 
+  /**
+   * Add task to a running (IN-PROGRESS) job or a job which has not started yet. Timeout for this
+   * operation is the default timeout which is 5 minutes. {@link TaskDriver#DEFAULT_TIMEOUT}
+   * Note1: Task cannot be added if the job is in an illegal state. A job can accept
+   * new task if the job is in-progress or it has not started yet.
+   * Note2: The job can only be added to non-targeted jobs.
+   * Note3: The taskID for the new task should be unique. If not, this API throws an exception.
+   * Note4: In case of timeout exception, it is the user's responsibility to check whether the task
+   * has been successfully added or not.
+   * @param workflowName
+   * @param jobName
+   * @param taskConfig
+   * @throws Exception if there is an issue with the request or the operation. 1-
+   *           IllegalArgumentException will be thrown if the inputs are invalid. 2- HelixException
+   *           will be thrown if the job is not in the states to accept a new task or if there is
+   *           any issue in updating jobConfig. 3- TimeoutException will be thrown if the outcome of
+   *           the task addition is unknown and cannot be verified.
+   */
+  public void addTask(String workflowName, String jobName, TaskConfig taskConfig) throws Exception {
+    addTask(workflowName, jobName, taskConfig, DEFAULT_TIMEOUT);
+  }
+
+  /**
+   * Add task to a running (IN-PROGRESS) job or a job which has not started yet
+   * Note1: Task cannot be added if the job is in an illegal state. A job can accept
+   * new task if the job is in-progress or it has not started yet.
+   * Note2: The job can only be added to non-targeted jobs.
+   * Note3: The taskID for the new task should be unique. If not, this API throws an exception.
+   * Note4: In case of timeout exception, it is the user's responsibility to check whether the task
+   * has been successfully added or not.
+   * Note5: timeout is the time that this API checks whether the task has been successfully added or
+   * not.
+   * @param workflowName
+   * @param jobName
+   * @param taskConfig
+   * @param timeoutMs
+   * @throws Exception if there is an issue with the request or the operation. 1-
+   *           IllegalArgumentException will be thrown if the inputs are invalid. 2- HelixException
+   *           will be thrown if the job is not in the states to accept a new task or if there is
+   *           any issue in updating jobConfig. 3- TimeoutException will be thrown if the outcome of
+   *           the task addition is unknown and cannot be verified.
+   */
+  public void addTask(String workflowName, String jobName, TaskConfig taskConfig, long timeoutMs)
+      throws Exception {
+
+    if (timeoutMs < DEFAULT_SLEEP) {
+      throw new IllegalArgumentException(
+          String.format("Timeout is less than the minimum acceptable timeout value which is %s ms",
+              DEFAULT_SLEEP));
+    }
+
+    long endTime = System.currentTimeMillis() + timeoutMs;
+
+    validateAddTaskConfigs(workflowName, jobName, taskConfig);
+
+    String nameSpaceJobName = TaskUtil.getNamespacedJobName(workflowName, jobName);
+    WorkflowContext workflowContext = getWorkflowContext(workflowName);
+    JobContext jobContext = getJobContext(nameSpaceJobName);
+    if (workflowContext == null || jobContext == null) {
+      // Workflow context or job context is null. It means job has not been started. Hence task can
+      // be added to the job
+      addTaskToJobConfig(workflowName, jobName, taskConfig, endTime);
+      return;
+    }
+
+    TaskState jobState = workflowContext.getJobState(nameSpaceJobName);
+
+    if (jobState == null) {
+      // Null job state means the job has not started yet
+      addTaskToJobConfig(workflowName, jobName, taskConfig, endTime);
+      return;
+    }
+
+    if (ILLEGAL_JOB_STATES_FOR_TASK_MODIFICATION.contains(jobState)) {
+      throw new HelixException(
+          String.format("Job %s is in illegal state to accept new task. Job State is %s",
+              nameSpaceJobName, jobState));
+    }
+    addTaskToJobConfig(workflowName, jobName, taskConfig, endTime);
+  }
+
+  /**
+   * The helper method which check the workflow, job and task configs to determine if new task can
+   * be added to the job
+   * @param workflowName
+   * @param jobName
+   * @param taskConfig
+   */
+  private void validateAddTaskConfigs(String workflowName, String jobName, TaskConfig taskConfig) {
+    WorkflowConfig workflowConfig = TaskUtil.getWorkflowConfig(_accessor, workflowName);
+    String nameSpaceJobName = TaskUtil.getNamespacedJobName(workflowName, jobName);
+    JobConfig jobConfig = TaskUtil.getJobConfig(_accessor, nameSpaceJobName);
+
+    if (workflowConfig == null) {
+      throw new IllegalArgumentException(
+          String.format("Workflow config for workflow %s does not exist!", workflowName));
+    }
+
+    if (jobConfig == null) {
+      throw new IllegalArgumentException(
+          String.format("Job config for job %s does not exist!", nameSpaceJobName));
+    }
+
+    if (taskConfig == null) {
+      throw new IllegalArgumentException("TaskConfig is null!");
+    }
+
+    if (taskConfig.getId() == null) {
+      throw new HelixException("Task cannot be added because taskID is null!");
+    }
+
+    if (jobConfig.getTargetResource() != null) {
+      throw new HelixException(String.format(
+          "Job %s is a targeted job. New task cannot be added to this job!", nameSpaceJobName));
+    }
+
+    if (taskConfig.getCommand() == null && jobConfig.getCommand() == null) {
+      throw new HelixException(
+          "Task cannot be added because both of the job and task have null command!");
+    }
+
+    if (taskConfig.getCommand() != null && jobConfig.getCommand() != null) {

Review comment:
       nit: could combine into (taskConfig.getCommand() == null != jobConfig.getCommand() == null) and say "Command must exist in either job or task, not both". 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org