You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@helix.apache.org by jx...@apache.org on 2017/06/23 20:54:20 UTC
helix git commit: Add task execution throttling manual into
TaskFramework document. Also update version to enable "mvn site".
Repository: helix
Updated Branches:
refs/heads/master e35fe4fff -> 91328cffe
Add task execution throttling manual into TaskFramework document.
Also update version to enable "mvn site".
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/91328cff
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/91328cff
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/91328cff
Branch: refs/heads/master
Commit: 91328cffe89d0b7b90a376d057668e5f5510359d
Parents: e35fe4f
Author: Jiajun Wang <jj...@linkedin.com>
Authored: Fri Jun 23 13:50:19 2017 -0700
Committer: Jiajun Wang <jj...@linkedin.com>
Committed: Fri Jun 23 13:52:11 2017 -0700
----------------------------------------------------------------------
website/0.6.1-incubating/pom.xml | 2 +-
website/0.6.2-incubating/pom.xml | 2 +-
website/0.6.3/pom.xml | 2 +-
website/0.6.4/pom.xml | 2 +-
website/0.6.5/pom.xml | 2 +-
website/0.6.6/pom.xml | 2 +-
website/0.6.7/pom.xml | 2 +-
website/0.6.8/pom.xml | 2 +-
.../site/markdown/tutorial_task_framework.md | 21 ++++++++++++++++++++
website/0.7.0-incubating/pom.xml | 2 +-
website/0.7.1/pom.xml | 2 +-
website/pom.xml | 2 +-
12 files changed, 32 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.1-incubating/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.1-incubating/pom.xml b/website/0.6.1-incubating/pom.xml
index 0b26f25..9697aa0 100644
--- a/website/0.6.1-incubating/pom.xml
+++ b/website/0.6.1-incubating/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.1-incubating-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.2-incubating/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.2-incubating/pom.xml b/website/0.6.2-incubating/pom.xml
index aa4eb0c..cad31bb 100644
--- a/website/0.6.2-incubating/pom.xml
+++ b/website/0.6.2-incubating/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.2-incubating-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.3/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.3/pom.xml b/website/0.6.3/pom.xml
index 97cba76..46efbb7 100644
--- a/website/0.6.3/pom.xml
+++ b/website/0.6.3/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.3-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.4/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.4/pom.xml b/website/0.6.4/pom.xml
index bc6e098..6e8d003 100644
--- a/website/0.6.4/pom.xml
+++ b/website/0.6.4/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.4-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.5/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.5/pom.xml b/website/0.6.5/pom.xml
index 5f266b3..2d16620 100644
--- a/website/0.6.5/pom.xml
+++ b/website/0.6.5/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.5-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.6/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.6/pom.xml b/website/0.6.6/pom.xml
index d6be30b..717c97b 100644
--- a/website/0.6.6/pom.xml
+++ b/website/0.6.6/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.6-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.7/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.7/pom.xml b/website/0.6.7/pom.xml
index c0654dc..3373f46 100644
--- a/website/0.6.7/pom.xml
+++ b/website/0.6.7/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.7-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.8/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.6.8/pom.xml b/website/0.6.8/pom.xml
index 59fcf2c..d7f3f09 100644
--- a/website/0.6.8/pom.xml
+++ b/website/0.6.8/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.6.8-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.6.8/src/site/markdown/tutorial_task_framework.md
----------------------------------------------------------------------
diff --git a/website/0.6.8/src/site/markdown/tutorial_task_framework.md b/website/0.6.8/src/site/markdown/tutorial_task_framework.md
index 2415a95..d809235 100644
--- a/website/0.6.8/src/site/markdown/tutorial_task_framework.md
+++ b/website/0.6.8/src/site/markdown/tutorial_task_framework.md
@@ -338,6 +338,27 @@ jobCfg.setInstanceGroupTag("INSTANCEGROUPTAG");
| _setIgnoreDependentJobFailure(boolean ignoreDependentJobFailure)_ | Set whether ignore the job failure of parent job of this job |
| _setJobType(String jobType)_ | Set the job type of this job |
+### Global Task Execution Throttling
+Besides the job option NumConcurrentTasksPerInstance, Helix also support global task execution throttling. This throttling is applied on the total number of running tasks on each participant.
+
+```
+InstanceConfig instanceConfig = HelixAdmin.getInstanceConfig(CLUSTER_NAME, PARTICIPANT);
+instanceConfig.setMaxConcurrentTask(perInstanceTaskLimitation);
+HelixAdmin.setInstanceConfig(CLUSTER_NAME, PARTICIPANT, instanceConfig);
+```
+
+Or, user can set MAX_CONCURRENT_TASK_PER_INSTANCE in ClusterConfig. The configuration will be applied to all instances in that cluster automatically.
+
+```
+Map<String, String> properties = new HashMap<String, String>();
+properties.put(ClusterConfig.ClusterConfigProperty.MAX_CONCURRENT_TASK_PER_INSTANCE.name(),
+ new Integer(perNodeTaskLimitation).toString());
+HelixAdmin.setConfig(clusterConfigScope, properties);
+```
+
+If user does not setup the global task throttling explicitly, each participant will have a default limitation based on the default task execution threadpool size.
+The tasks are throttled based on the jobs' start time. Helix controller will assign older jobs' tasks first.
+
### Monitor the status of your job
As we introduced the excellent util TaskDriver in Workflow Section, we have extra more functionality that provided to user. The user can synchronized wait Job or Workflow until it reaches certain STATES. The function Helix have API pollForJobState and pollForWorkflowState. For pollForJobState, it accepts arguments:
* Workflow name, required
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.7.0-incubating/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.7.0-incubating/pom.xml b/website/0.7.0-incubating/pom.xml
index 341aeba..43751f3 100644
--- a/website/0.7.0-incubating/pom.xml
+++ b/website/0.7.0-incubating/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.7.0-incubating-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/0.7.1/pom.xml
----------------------------------------------------------------------
diff --git a/website/0.7.1/pom.xml b/website/0.7.1/pom.xml
index d88223c..5a9fc8f 100644
--- a/website/0.7.1/pom.xml
+++ b/website/0.7.1/pom.xml
@@ -23,7 +23,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>website</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<artifactId>0.7.1-docs</artifactId>
http://git-wip-us.apache.org/repos/asf/helix/blob/91328cff/website/pom.xml
----------------------------------------------------------------------
diff --git a/website/pom.xml b/website/pom.xml
index c5b38ae..b17ef33 100644
--- a/website/pom.xml
+++ b/website/pom.xml
@@ -21,7 +21,7 @@ under the License.
<parent>
<groupId>org.apache.helix</groupId>
<artifactId>helix</artifactId>
- <version>0.7.2-SNAPSHOT</version>
+ <version>0.6.9-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>