You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tez.apache.org by je...@apache.org on 2016/08/15 15:06:08 UTC
tez git commit: TEZ-3360. Tez Custom Shuffle Handler Documentation
(jeagles)
Repository: tez
Updated Branches:
refs/heads/TEZ-3334 78eee17c6 -> 613f1e074
TEZ-3360. Tez Custom Shuffle Handler Documentation (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/613f1e07
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/613f1e07
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/613f1e07
Branch: refs/heads/TEZ-3334
Commit: 613f1e0743999c3bf558761c60165bcd2dcd05c7
Parents: 78eee17
Author: Jonathan Eagles <je...@yahoo-inc.com>
Authored: Mon Aug 15 10:05:26 2016 -0500
Committer: Jonathan Eagles <je...@yahoo-inc.com>
Committed: Mon Aug 15 10:05:26 2016 -0500
----------------------------------------------------------------------
TEZ-3334-CHANGES.txt | 1 +
docs/src/site/markdown/shuffle-handler.md | 72 ++++++++++++++++++++++++++
docs/src/site/markdown/user_guides.md | 1 +
docs/src/site/site.xml | 1 +
4 files changed, 75 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/TEZ-3334-CHANGES.txt
----------------------------------------------------------------------
diff --git a/TEZ-3334-CHANGES.txt b/TEZ-3334-CHANGES.txt
index 493614f..f93ec6e 100644
--- a/TEZ-3334-CHANGES.txt
+++ b/TEZ-3334-CHANGES.txt
@@ -4,6 +4,7 @@ Apache Tez Change Log
INCOMPATIBLE CHANGES:
ALL CHANGES:
+ TEZ-3360. Tez Custom Shuffle Handler Documentation
TEZ-3411. TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler port
TEZ-3412. Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString()
TEZ-3410. ShuffleHandler should use Path.SEPARATOR instead "/"
http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/shuffle-handler.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/shuffle-handler.md b/docs/src/site/markdown/shuffle-handler.md
new file mode 100644
index 0000000..5dc6289
--- /dev/null
+++ b/docs/src/site/markdown/shuffle-handler.md
@@ -0,0 +1,72 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Shuffle Handler</title></head>
+
+## Tez Shuffle Handler Overview
+
+A Tez specific shuffle handler allows Tez DAGs to shuffle data in a way that takes advantage of the new features in Tez.
+In particular, the Tez shuffle handler allows DAGs to shuffle data more efficiently for Tez\u2019s new data movements types
+and runtime optimizations, such as auto-reduce parallelism. Long running Tez sessions will be able to clean up
+intermediate data for completed queries and Tez applications can decide to clean up completed intermediate data for
+running applications.
+
+## Setup for the Tez Shuffle Handler
+---------
+
+_Requires_: **Apache Tez 0.9.0 or above**
+
+Configuration in the client specify the Tez shuffle handler
+
+```
+tez-site.xml
+-------------
+...
+<property>
+ <name>tez.am.shuffle.auxiliary-service.id</name>
+ <value>tez_shuffle</value>
+</property>
+...
+
+```
+
+## Deploying the Tez Shuffle Handler
+
+The Tez Shuffle Handler jar artifact org.apache.org:tez-aux-services needs to be placed into the Node Manager classpath and restarted
+
+## Setup for Node Manager
+
+_Requires_: **Apache Hadoop 2.6.0 or above**
+
+The following configuration needs to be setup in the Node Manager yarn-site.xml to enable the Tez Shuffle Handler
+
+```
+yarn-site.xml
+-------------
+...
+<property>
+ <name>yarn.nodemanager.aux-services</name>
+ <value>tez_shuffle</value>
+</property>
+
+<property>
+ <name>yarn.nodemanager.aux-services.tez_shuffle.class</name>
+ <value>org.apache.tez.auxservices.ShuffleHandler</value>
+</property>
+...
+
+```
http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/user_guides.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/user_guides.md b/docs/src/site/markdown/user_guides.md
index 890974e..453df16 100644
--- a/docs/src/site/markdown/user_guides.md
+++ b/docs/src/site/markdown/user_guides.md
@@ -23,4 +23,5 @@
- [Access Control in Tez](./tez_acls.html)
- [Embedding Application Specific Data into Tez UI](./tez_ui_user_data.html)
- [Tez UI - Overview and installation](./tez-ui.html)
+ - [Tez Shuffle Handler - Overview and installation](./shuffle-handler.html)
http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/site.xml
----------------------------------------------------------------------
diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml
index 3317bac..5674951 100644
--- a/docs/src/site/site.xml
+++ b/docs/src/site/site.xml
@@ -107,6 +107,7 @@
<item name="Install Guide" href="install.html"/>
<item name="Local Mode" href="localmode.html"/>
<item name="Tez UI" href="tez-ui.html"/>
+ <item name="Tez Shuffle Handler" href="shuffle-handler.html"/>
<item name="User Guides" href="user_guides.html"/>
</menu>