You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@helix.apache.org by lx...@apache.org on 2018/03/22 18:17:48 UTC
helix git commit: Tutorial update for task framework new features in
0.8
Repository: helix
Updated Branches:
refs/heads/master e1176fe40 -> df012d5b8
Tutorial update for task framework new features in 0.8
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/df012d5b
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/df012d5b
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/df012d5b
Branch: refs/heads/master
Commit: df012d5b8782a11e9e9a01bd25f81bdfb5c8607d
Parents: e1176fe
Author: jiajunwang <er...@gmail.com>
Authored: Mon Mar 19 17:22:31 2018 -0700
Committer: Lei Xia <lx...@linkedin.com>
Committed: Thu Mar 22 11:17:20 2018 -0700
----------------------------------------------------------------------
website/0.8.0/src/site/markdown/Tutorial.md | 2 +
.../site/markdown/tutorial_task_framework.md | 25 +++++++-
.../site/markdown/tutorial_task_throttling.md | 41 ++++++++++++
.../markdown/tutorial_user_content_store.md | 67 ++++++++++++++++++++
4 files changed, 134 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/helix/blob/df012d5b/website/0.8.0/src/site/markdown/Tutorial.md
----------------------------------------------------------------------
diff --git a/website/0.8.0/src/site/markdown/Tutorial.md b/website/0.8.0/src/site/markdown/Tutorial.md
index 75fcc5b..d487d99 100644
--- a/website/0.8.0/src/site/markdown/Tutorial.md
+++ b/website/0.8.0/src/site/markdown/Tutorial.md
@@ -50,6 +50,8 @@ Convention: we first cover the _basic_ approach, which is the easiest to impleme
12. [YAML Cluster Setup](./tutorial_yaml.html)
13. [Helix Agent (for non-JVM systems)](./tutorial_agent.html)
14. [Task Framework](./tutorial_task_framework.html)
+ 1. [Task with User Defined Content Store](./tutorial_user_content_store.html)
+ 2. [Task Throttling](./tutorial_task_throttling.html)
15. [Helix REST Service 2.0](./tutorial_rest_service.html)
16. [Helix UI Setup](./tutorial_ui.html)
http://git-wip-us.apache.org/repos/asf/helix/blob/df012d5b/website/0.8.0/src/site/markdown/tutorial_task_framework.md
----------------------------------------------------------------------
diff --git a/website/0.8.0/src/site/markdown/tutorial_task_framework.md b/website/0.8.0/src/site/markdown/tutorial_task_framework.md
index 2415a95..9659ada 100644
--- a/website/0.8.0/src/site/markdown/tutorial_task_framework.md
+++ b/website/0.8.0/src/site/markdown/tutorial_task_framework.md
@@ -183,7 +183,7 @@ taskDriver.resume(myWorkflow);
#### Delete a Workflow
-Simliar to start, stop and resume, delete operation is supported by TaskDriver.
+Similar to start, stop and resume, delete operation is supported by TaskDriver.
```
taskDriver.delete(myWorkflow);
@@ -205,6 +205,14 @@ Jobs can have dependencies. If one job2 depends job1, job2 will not be scheduled
myWorkflowBuilder.addParentChildDependency(ParentJobName, ChildJobName);
```
+#### Schedule a workflow for executing in a future time
+
+Application can create a workflow with a ScheduleConfig so as to schedule it to be executed in a future time.
+
+```
+myWorkflowBuilder.setScheduleConfig(ScheduleConfig.oneTimeDelayedStart(new Date(inFiveSeconds)));
+```
+
#### Additional Workflow Options
| Additional Config Options | Detail |
@@ -321,6 +329,19 @@ To add instance group tag, just set it in JobConfig.Builder:
jobCfg.setInstanceGroupTag("INSTANCEGROUPTAG");
```
+#### Delayed scheduling job
+
+Set up a schedule plan for the job.
+If both items are set, Helix will calculate and use the later one.
+
+```
+myJobCfgBuilder.setExecutionDelay(delayMs);
+myJobCfgBuilder.setExecutionStart(startTimeMs);
+```
+
+Note that the scheduled job needs to be runnable first. Then Helix will start checking it's configuration for scheduling.
+If any parent jobs are not finished, the job won't be scheduled even the scheduled timestamp has already passed.
+
#### Additional Job Options
| Operation | Detail |
@@ -337,6 +358,8 @@ jobCfg.setInstanceGroupTag("INSTANCEGROUPTAG");
| _setTaskRetryDelay(long v)_ | Set the delay time before a task retry |
| _setIgnoreDependentJobFailure(boolean ignoreDependentJobFailure)_ | Set whether ignore the job failure of parent job of this job |
| _setJobType(String jobType)_ | Set the job type of this job |
+| _setExecutionDelay(String delay)_ | Set the delay time to schedule job execution |
+| _setExecutionStart(String start)_ | Set the start time to schedule job execution |
### Monitor the status of your job
As we introduced the excellent util TaskDriver in Workflow Section, we have extra more functionality that provided to user. The user can synchronized wait Job or Workflow until it reaches certain STATES. The function Helix have API pollForJobState and pollForWorkflowState. For pollForJobState, it accepts arguments:
http://git-wip-us.apache.org/repos/asf/helix/blob/df012d5b/website/0.8.0/src/site/markdown/tutorial_task_throttling.md
----------------------------------------------------------------------
diff --git a/website/0.8.0/src/site/markdown/tutorial_task_throttling.md b/website/0.8.0/src/site/markdown/tutorial_task_throttling.md
new file mode 100644
index 0000000..e9029d9
--- /dev/null
+++ b/website/0.8.0/src/site/markdown/tutorial_task_throttling.md
@@ -0,0 +1,41 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<head>
+ <title>Tutorial - Task Throttling</title>
+</head>
+
+## [Helix Tutorial](./Tutorial.html): Task Throttling
+
+In this chapter, we\'ll learn how to control the parallel execution of tasks in the task framework.
+
+### Task Throttling Configuration
+
+Helix can control the number of tasks that are executed in parallel according to multiple thresholds.
+Applications can set these thresholds in the following configuration items:
+
+* JobConfig.ConcurrentTasksPerInstance The number of concurrent tasks in this job that are allowed to run on an instance.
+* InstanceConfig.MAX_CONCURRENT_TASK The number of total concurrent tasks that are allowed to run on an instance.
+
+Also see [WorkflowConfig.ParallelJobs](./tutorial_task_framework.html).
+
+### Job Priority for Task Throttling
+
+Whenever there are too many tasks to be scheduled according to the threshold, Helix will prioritize the older jobs.
+The age of a job is calculated based on the job start time.
http://git-wip-us.apache.org/repos/asf/helix/blob/df012d5b/website/0.8.0/src/site/markdown/tutorial_user_content_store.md
----------------------------------------------------------------------
diff --git a/website/0.8.0/src/site/markdown/tutorial_user_content_store.md b/website/0.8.0/src/site/markdown/tutorial_user_content_store.md
new file mode 100644
index 0000000..81c502b
--- /dev/null
+++ b/website/0.8.0/src/site/markdown/tutorial_user_content_store.md
@@ -0,0 +1,67 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<head>
+ <title>Tutorial - User Defined Content Store for Tasks</title>
+</head>
+
+## [Helix Tutorial](./Tutorial.html): User Defined Content Store for Tasks
+
+The purpose of user defined content store is to provide an easy use feature for some task dedicated meta temporary store.
+In this chapter, we\'ll learn how to implement and use content store in the user defined tasks.
+
+### Content Store Implementation
+
+Extends abstract class UserContentStore.
+
+ private static class ContentStoreTask extends UserContentStore implements Task {
+ @Override public TaskResult run() {
+ ...
+ }
+ @Override public void cancel() {
+ ...
+ }
+ }
+
+The default methods support 3 types of scopes:
+1. WORKFLOW: Define the content store in workflow level
+2. JOB: Define the content store in job level
+3. TASK: Define the content store in task level
+
+### Content Store Usage
+
+Access content store in Task.run() method.
+
+ private static class ContentStoreTask extends UserContentStore implements Task {
+ @Override public TaskResult run() {
+ // put values into the store
+ putUserContent("ContentTest", "Value1", Scope.JOB);
+ putUserContent("ContentTest", "Value2", Scope.WORKFLOW);
+ putUserContent("ContentTest", "Value3", Scope.TASK);
+
+ // get the values with the same key in the different scopes
+ if (!getUserContent("ContentTest", Scope.JOB).equals("Value1") ||
+ !getUserContent("ContentTest", Scope.WORKFLOW).equals("Value2") ||
+ !getUserContent("ContentTest", Scope.TASK).equals("Value3")) {
+ return new TaskResult(TaskResult.Status.FAILED, null);
+ }
+
+ return new TaskResult(TaskResult.Status.COMPLETED, null);
+ }
+ }