You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tez.apache.org by je...@apache.org on 2014/09/03 22:24:27 UTC
git commit: TEZ-1310. Update website documentation framework (jeagles)
Repository: tez
Updated Branches:
refs/heads/master 6b28417e8 -> 119d5c791
TEZ-1310. Update website documentation framework (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/tez/repo
Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/119d5c79
Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/119d5c79
Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/119d5c79
Branch: refs/heads/master
Commit: 119d5c791006b2247ffb80b088b0961963b2e05f
Parents: 6b28417
Author: Jonathan Eagles <je...@gmail.com>
Authored: Wed Sep 3 15:24:01 2014 -0500
Committer: Jonathan Eagles <je...@gmail.com>
Committed: Wed Sep 3 15:24:01 2014 -0500
----------------------------------------------------------------------
INSTALL.md | 1 +
INSTALL.txt | 80 -----------------
docs/src/site/apt/index.apt | 54 ------------
docs/src/site/apt/install.apt | 98 ---------------------
docs/src/site/apt/install_0_5_0.apt | 118 --------------------------
docs/src/site/apt/localmode.apt | 89 -------------------
docs/src/site/apt/privacy-policy.apt | 55 ------------
docs/src/site/apt/talks.apt | 31 -------
docs/src/site/markdown/index.md | 47 ++++++++++
docs/src/site/markdown/install.md | 109 ++++++++++++++++++++++++
docs/src/site/markdown/install_0_5_0.md | 117 +++++++++++++++++++++++++
docs/src/site/markdown/localmode.md | 107 +++++++++++++++++++++++
docs/src/site/markdown/privacy-policy.md | 52 ++++++++++++
docs/src/site/markdown/talks.md | 32 +++++++
pom.xml | 2 +-
15 files changed, 466 insertions(+), 526 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/INSTALL.md
----------------------------------------------------------------------
diff --git a/INSTALL.md b/INSTALL.md
new file mode 120000
index 0000000..e9f9623
--- /dev/null
+++ b/INSTALL.md
@@ -0,0 +1 @@
+docs/src/site/markdown/install.md
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/INSTALL.txt
----------------------------------------------------------------------
diff --git a/INSTALL.txt b/INSTALL.txt
deleted file mode 100644
index 887d423..0000000
--- a/INSTALL.txt
+++ /dev/null
@@ -1,80 +0,0 @@
-How to use TEZ
-=======================
-
-Tez provides an ApplicationMaster that can run any arbritary DAG of tasks. It also
-provides a translation layer to run MR jobs using the MR APIs. This translation
-layer is not fully feature compatible so if you do see any issues with running your
-existing MR jobs on TEZ, please file jiras.
-
-Install/Deploy Instructions
-===========================
-
-1) Deploy Apache Hadoop either using the 2.2.0 release or a compatible 2.x version.
-2) Build tez using "mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true"
- - If you prefer to run the unit tests, remove skipTests from the command above.
- - A tarball containing the libraries required to run tez will be created at tez-dist/target/tez-0.6.0-SNAPSHOT.tar.gz
-3) Copy the relevant tez tarball into HDFS, and configure tez-site.xml
- - A tez tarball containing tez and hadoop libraries will be found at tez-dist/target/tez-0.6.0-SNAPSHOT.tar.gz
- - Assuming that the tez jars are put in /apps/ on HDFS, the command would be
- "hadoop fs -mkdir /apps/tez-0.6.0-SNAPSHOT"
- "hadoop fs -copyFromLocal tez-dist/target/tez-0.6.0-SNAPSHOT.tar.gz /apps/tez-0.6.0-SNAPSHOT/"
- - tez-site.xml configuration
- - Set tez.lib.uris to point to the tar.gz uploaded to HDFS. Assuming the steps mentioned so far were followed,
- set tez.lib.uris to "${fs.defaultFS}/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0-SNAPSHOT.tar.gz"
- - Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml, or if it is set, the value should be false
-4) Optional: If running existing MapReduce jobs on Tez. Modify mapred-site.xml to change
-"mapreduce.framework.name" property from its default value of "yarn" to "yarn-tez"
-5) Configure the client node to include the tez-libraries in the hadoop classpath
- - Extract the tez tarball created in step 2 to a local directory - (assuming TEZ_JARS is where the files will be decompressed for the next steps)
- "tar -xvzf tez-dist/target/tez-0.6.0-SNAPSHOT.tar.gz -C $TEZ_JARS"
- - set HADOOP_CLASSPATH to include the tez-libraries
- - set TEZ_CONF_DIR to the location of tez-site.xml
- - The command to set up the classpath should be something like:
- "export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*"
- - Please note the "*" which is an important requirement when setting up classpaths for directories containing jar files.
-
-6) There is a basic example of a Tez job in the tez-examples.jar. Refer to OrderedWordCount.java
-in the source code. To run this example:
-
-$HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount <input> <output>
-
-This will use the TEZ DAG ApplicationMaster to run the ordered word count job. This job is similar
-to the word count example except that it also orders all words based on the frequency of
-occurrence.
-
-There are multiple variations of orderedwordcount. You can take a look at TestOrderedWordCount.java
-in tez-tests for these variations. You can use it to run multiple
-DAGs serially on different inputs/outputs. These DAGs could be run separately as
-different applications or serially within a single TEZ session.
-
-$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
-
-The above will run multiple DAGs for each input-output pair. To use TEZ sessions,
-set -DUSE_TEZ_SESSION=true
-
-$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
-
-7) To test MR jobs you can submit an MR job as you normally would using something like:
-
-$HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-2.2.0-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
-
-This will use the TEZ DAG ApplicationMaster to run the MR job. This can be verified by looking at
-the AM's logs from the YARN ResourceManager UI. This needs mapred-site.xml to have "mapreduce.framework.name"
-set to "yarn-tez"
-
-
-Hadoop Installation dependent Install/Deploy Instructions
-=========================================================
-The above install instructions use Tez with pre-packaged Hadoop libraries included in the package and is the
-recommended method for installation. If its needed to make Tez use the existing cluster Hadoop libraries then
-follow this alternate machanism to setup Tez to use Hadoop libraries from the cluster.
-Step 3 above changes as follows. Also subsequent steps would use tez-dist/target/tez-0.6.0-SNAPSHOT-minimal.tar.gz instead of tez-dist/target/tez-0.6.0-SNAPSHOT.tar.gz
- - A tez build without Hadoop dependencies will be available at tez-dist/target/tez-0.6.0-SNAPSHOT-minimal.tar.gz
- - Assuming that the tez jars are put in /apps/ on HDFS, the command would be
- "hadoop fs -mkdir /apps/tez-0.6.0-SNAPSHOT"
- "hadoop fs -copyFromLocal tez-dist/target/tez-0.6.0-SNAPSHOT-minimal.tar.gz /apps/tez-0.6.0-SNAPSHOT"
- - tez-site.xml configuration
- - Set tez.lib.uris to point to the paths in HDFS containing the tez jars. Assuming the steps mentioned so far were followed,
- set tez.lib.uris to "${fs.defaultFS}/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0-SNAPSHOT-minimal.tar.gz
- - set tez.use.cluster.hadoop-libs to true
-
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/index.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/index.apt b/docs/src/site/apt/index.apt
deleted file mode 100644
index 2e467b8..0000000
--- a/docs/src/site/apt/index.apt
+++ /dev/null
@@ -1,54 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-~~
- ------
- Welcome to Apache Tez
- ------
-
-Introduction
-
- The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop {{{http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html}Apache Hadoop YARN}}
-
- The 2 main design themes for Tez are:
-
- * <<Empowering end users by:>>
-
- * Expressive dataflow definition APIs
-
- * Flexible Input-Processor-Output runtime model
-
- * Data type agnostic
-
- * Simplifying deployment
-
-
- * <<Execution Performance>>
-
- * Performance gains over Map Reduce
-
- * Optimal resource management
-
- * Plan reconfiguration at runtime
-
- * Dynamic physical data flow decisions
-
- []
-
- By allowing projects like Apache Hive and Apache Pig to run a complex DAG of tasks, Tez can be used to process data, that earlier took multiple MR jobs, now in a single Tez job as shown below.
-
-[./images/PigHiveQueryOnMR.png] Flow for a Hive or Pig Query on MapReduce
-
-[./images/PigHiveQueryOnTez.png] Flow for a Hive or Pig Query on Tez
-
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/install.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/install.apt b/docs/src/site/apt/install.apt
deleted file mode 100644
index 6b4af8a..0000000
--- a/docs/src/site/apt/install.apt
+++ /dev/null
@@ -1,98 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-~~
- -----
- Install and Deployment Instructions
- -----
-
-{{{./install_0_5_0.html}Install instructions for Tez-0.5.0-SNAPSHOT - master branch}}
-
-Install/Deploy Instructions for the latest Tez release {{{http://www.apache.org/dyn/closer.cgi/incubator/tez/tez-0.4.1-incubating/}(Tez-0.4.1 src)}}
-
- [[i]] Deploy Apache Hadoop using either the 2.2.0 release or a compatible 2.x version.
-
- * One thing to note though when compiling Tez is that you will need to change the value of the hadoop.version property in the top-level pom.xml to match the version of the hadoop branch being used.
-
- [[i]] Build tez using "mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true"
-
- * This assumes that you have already installed JDK6 or later, Maven 3 or later and Protocol Buffers (protoc compiler) 2.5 or later
-
- * If you prefer to run the unit tests, remove skipTests from the command above.
-
- * If you would like to create a tarball of the release, use "mvn clean package -Dtar -DskipTests=true -Dmaven.javadoc.skip=true"
-
- * If you use Eclipse IDE, you can import the peojects using "Import/Maven/Existing Maven Projects". Eclipse does not automatically generate Java sources or include the generated sources into the projects. Please build using maven as described above and then use Project Properties to include "target/generated-sources/java" as a source directory into the "Java Build Path" for these projects: tez-api, tez-mapreduce, tez-runtime-internals and tez-runtime-library. This needs to be done just once after importing the project.
-
- [[i]] Copy the tez jars and their dependencies into HDFS.
-
- * The tez jars and dependencies will be found in tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating if you run the intial command mentioned in step 2.
-
- * Assuming that the tez jars are put in /apps/ on HDFS, the command would be "hadoop dfs -put tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating /apps/"
-
- * Please do not upload the tarball to HDFS, upload only the jars.
-
-
- [[i]] Configure tez-site.xml to set tez.lib.uris to point to the paths in HDFS containing the jars. Please note that the paths are not searched recursively so for <basedir> and <basedir>/lib/, you will need to configure the 2 paths as a comma-separated list.
- * Assuming you followed step 3, the value would be: "$\{fs.default.name\}/apps/tez-0.4.1-incubating,$\{fs.default.name\}/apps/tez-0.4.1-incubating/lib/"
-
- [[i]] Modify mapred-site.xml to change "mapreduce.framework.name" property from its default value of "yarn" to "yarn-tez"
-
- [[i]] Set HADOOP_CLASSPATH to have the following paths in it:
-
- * TEZ_CONF_DIR - location of tez-site.xml
-
- * TEZ_JARS and TEZ_JARS/libs - location of the tez jars and dependencies.
-
- * The command to set up the classpath should be something like: "export HADOOP_CLASSPATH=$\{TEZ_CONF_DIR\}:$\{TEZ_JARS\}/*:$\{TEZ_JARS\}/lib/*". Please note the "*" which is an important requirement when setting up classpaths for directories containing jar files.
-
-
- [[i]] Submit a MR job as you normally would using something like:
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
-
-+---+
-
- This will use the TEZ DAG ApplicationMaster to run the MR job. This can be verified by looking at the AM's logs from the YARN ResourceManager UI.
-
- [[i]] There is a basic example of using an MRR job in the tez-mapreduce-examples.jar. Refer to OrderedWordCount.java
-in the source code. To run this example:
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input> <output>
-
-+---+
-
- This will use the TEZ DAG ApplicationMaster to run the ordered word count job. This job is similar to the word count example except that it also orders all words based on the frequency of occurrence.
-
- There are multiple variations to run orderedwordcount. You can use it to run multiple DAGs serially on different inputs/outputs. These DAGs could be run separately as different applications or serially within a single TEZ session.
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
-
-+---+
-
- The above will run multiple DAGs for each input-output pair.
-
- To use TEZ sessions, set -DUSE_TEZ_SESSION=true
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
-
-+---+
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/install_0_5_0.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/install_0_5_0.apt b/docs/src/site/apt/install_0_5_0.apt
deleted file mode 100644
index 0cabeaa..0000000
--- a/docs/src/site/apt/install_0_5_0.apt
+++ /dev/null
@@ -1,118 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-~~
- -----
- Install and Deployment Instructions
- -----
-
-Install/Deploy Instructions for Tez-current (0.5.0-SNAPSHOT, branch master)
-
- [[i]] Deploy Apache Hadoop using either the 2.2.0 release or a compatible 2.x version.
-
- * One thing to note though when compiling Tez is that you will need to change the value of the hadoop.version property in the top-level pom.xml to match the version of the hadoop branch being used.
-
- [[i]] Build tez using "mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true"
-
- * This assumes that you have already installed JDK6 or later, Maven 3 or later and Protocol Buffers (protoc compiler) 2.5 or later
-
- * If you prefer to run the unit tests, remove skipTests from the command above.
-
- * If you use Eclipse IDE, you can import the peojects using "Import/Maven/Existing Maven Projects". Eclipse does not automatically generate Java sources or include the generated sources into the projects. Please build using maven as described above and then use Project Properties to include "target/generated-sources/java" as a source directory into the "Java Build Path" for these projects: tez-api, tez-mapreduce, tez-runtime-internals and tez-runtime-library. This needs to be done just once after importing the project.
-
- [[i]] Copy the relevant tez tarball into HDFS, and configure tez-site.xml
-
- * A tez tarball containing tez and hadoop libraries will be found at tez-dist/target/tez-0.5.0-SNAPSHOT.tar.gz
-
- * Assuming that the tez jars are put in /apps/ on HDFS, the command would be
-
-+--+
-
- "hadoop dfs -mkdir /apps/tez-0.5.0-SNAPSHOT"
- "hadoop dfs -copyFromLocal tez-dist/target/tez-0.5.0-SNAPSHOT-archive.tar.gz /apps/tez-0.5.0-SNAPSHOT/"
-
-+--+
-
- * tez-site.xml configuration.
-
- * Set tez.lib.uris to point to the tar.gz uploaded to HDFS. Assuming the steps mentioned so far were followed,
-
-+--+
-
- set tez.lib.uris to "${fs.default.name}/apps/tez-0.5.0-SNAPSHOT/tez-0.5.0-SNAPSHOT.tar.gz"
-
-+--+
- * Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml, or if it is set, the value should be false
-
- [[i]] Optional: If running existing MapReduce jobs on Tez. Modify mapred-site.xml to change "mapreduce.framework.name" property from its default value of "yarn" to "yarn-tez"
-
- [[i]] Configure the client node to include the tez-libraries in the hadoop classpath
-
- * Extract the tez tarball created in step 2 to a local directory - (assuming TEZ_JARS is where the files will be decompressed for the next steps)
-
-+--+
-
-"tar -xvzf tez-dist/target/tez-0.5.0-SNAPSHOT.tar.gz -C $TEZ_JARS"
-
-+--+
- * set TEZ_CONF_DIR to the location of tez-site.xml
-
- * The command to set up the classpath should be something like:
-
-+--+
-
-"export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*"
-
-+--+
-
- * Please note the "*" which is an important requirement when setting up classpaths for directories containing jar files.
-
- [[i]] Submit a MR job as you normally would using something like:
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
-
-+---+
-
- This will use the TEZ DAG ApplicationMaster to run the MR job. This can be verified by looking at the AM's logs from the YARN ResourceManager UI.
-
- [[i]] There is a basic example of using an MRR job in the tez-examples.jar. Refer to OrderedWordCount.java
-in the source code. To run this example:
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount <input> <output>
-
-+---+
-
- This will use the TEZ DAG ApplicationMaster to run the ordered word count job. This job is similar to the word count example except that it also orders all words based on the frequency of occurrence.
-
- Tez DAGs could be run separately as different applications or serially within a single TEZ session. There is a different variation of orderedwordcount in tez-tests that supports the use of Sessions and handling multiple input-output pairs. You can use it to run multiple DAGs serially on different inputs/outputs.
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
-
-+---+
-
- The above will run multiple DAGs for each input-output pair.
-
- To use TEZ sessions, set -DUSE_TEZ_SESSION=true
-
-+---+
-
-$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
-
-+---+
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/localmode.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/localmode.apt b/docs/src/site/apt/localmode.apt
deleted file mode 100644
index 25fe13a..0000000
--- a/docs/src/site/apt/localmode.apt
+++ /dev/null
@@ -1,89 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-~~
------
- Tez Local Mode
------
-
- Tez Local Mode is a development tool to test Tez jobs without needing to bring up a Hadoop cluster. Local Mode runs the Tez compoenents - AppMaster, TaskRunner that are used when executing a job on a cluster. From a developer tool perspective, it offers several advantages.
-
- * Fast prototyping - Hadoop setup, launch cost etc not involved.
-
- * Unit testing - Fast execution since the overhead of allocating resources, launching JVMs etc is removed.
-
- * Easy debuggability - Single JVM running all user code.
-
-
- While majority of the components are re-used in Local Mode, there are some bits which are not
-
- * Scheduling and Container Re-Use differs.
-
- * Handling of YARN Local Resources. Local Mode expects necessary jars to be loaded with the Client when executing.
-
- * Contains some performance improvements - like skipping RPC invocations since everything runs within the same JVM.
-
-
- Running a DAG in Local Mode
-
- * "tez.local.mode" should be set to true in the confgiuration instance used to create the TezClient.
-
- * The FileSystem must be configured to the local file system ("fs.defaultFS" must be set to "file:///"). This is required to be setup in all Configuration instances used to create a DAG. Typically, when using Tez for testing and prototyping without a Hadoop cluster, this is not a problem. It becomes a problem when Hadoop Configuration files are in the classpath, with a different default filesystem configured.
-
- * Setup the fetchers to make use of local reads instead of fetching from remote nodes. ("tez.runtime.optimize.local.fetch must be set to true)
-
- * Beyond this, no other changes are required, to make use of Local Mode instead of running a job on a cluster.
-
- * If using this in code, the following changes should be made to configuration, after which this configuration instance becomes the base for all other Configuration instances.
-
-+------------------
-Configuration conf = new Configuration();
-conf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
-conf.set("fs.defaultFS", "file:///");
-conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH, true);
-+------------------
-
- * If using a tez-site.xml config file, it should contain the following entries
-
-+------------------
-<property>
- <name>fs.defaultFS</name>
- <value>file:///</value>
-</property>
-<property>
- <name>tez.local.mode</name>
- <value>true</value>
-</property>
-<property>
- <name>tez.runtime.optimize.local.fetch</name>
- <value>true</value>
-</property>
-+------------------
-
- Things to watch out for
-
- * In current Local Mode, large amount of input data may lead to JVM out of memory since all TEZ components are running in single JVM. The input data size should be kept small.
-
- * TezConfiguration.TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS(tez.am.inline.task.execution.max-tasks) should not be changed (defaults to 1).
-
- * "tez.history.logging.service.class" should be the default value: "org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService". It means ATS is disabbled in current Local Mode.
-
- Potential pitfalls when moving from Local Mode to a real cluster
-
- * Resource requirements (CPU, Memory, etc) which would otherwise have been specified for a YARN Cluster will now start taking affect, and should be considered.
-
- * The Java Options and Environment variables which may have been setup for the DAG do not take affect in Local Mode, and could be a source of migration problems.
-
- * The ObjectRegistry will work within a single task, when running in Local Mode. The behaviour would be different on a real cluster, where it would work across tasks which share the same container.
-
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/privacy-policy.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/privacy-policy.apt b/docs/src/site/apt/privacy-policy.apt
deleted file mode 100644
index a683d54..0000000
--- a/docs/src/site/apt/privacy-policy.apt
+++ /dev/null
@@ -1,55 +0,0 @@
- ----
- Privacy Policy
- -----
- Olivier Lamy
- -----
- 2012-06-18
- -----
-
-~~ Licensed to the Apache Software Foundation (ASF) under one
-~~ or more contributor license agreements. See the NOTICE file
-~~ distributed with this work for additional information
-~~ regarding copyright ownership. The ASF licenses this file
-~~ to you under the Apache License, Version 2.0 (the
-~~ "License"); you may not use this file except in compliance
-~~ with the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing,
-~~ software distributed under the License is distributed on an
-~~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-~~ KIND, either express or implied. See the License for the
-~~ specific language governing permissions and limitations
-~~ under the License.
-
-~~ NOTE: For help with the syntax of this file, see:
-~~ http://maven.apache.org/doxia/references/apt-format.html
-
-Privacy Policy
-
- Information about your use of this website is collected using server access logs and a tracking cookie. The
- collected information consists of the following:
-
- [[1]] The IP address from which you access the website;
-
- [[2]] The type of browser and operating system you use to access our site;
-
- [[3]] The date and time you access our site;
-
- [[4]] The pages you visit; and
-
- [[5]] The addresses of pages from where you followed a link to our site.
-
- []
-
- Part of this information is gathered using a tracking cookie set by the
- {{{http://www.google.com/analytics/}Google Analytics}} service and handled by Google as described in their
- {{{http://www.google.com/privacy.html}privacy policy}}. See your browser documentation for instructions on how to
- disable the cookie if you prefer not to share this data with Google.
-
- We use the gathered information to help us make our site more useful to visitors and to better understand how and
- when our site is used. We do not track or collect personally identifiable information or associate gathered data
- with any personally identifying information from other sources.
-
- By using this website, you consent to the collection of this data in the manner and for the purpose described above.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/apt/talks.apt
----------------------------------------------------------------------
diff --git a/docs/src/site/apt/talks.apt b/docs/src/site/apt/talks.apt
deleted file mode 100644
index 9903f0a..0000000
--- a/docs/src/site/apt/talks.apt
+++ /dev/null
@@ -1,31 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-~~
- ----
- Presentations and Talks on Tez
- ----
-
-Talks
-
- * Apache Tez : Accelerating Hadoop Query Processing by Arun Murthy and Bikas Saha at {{{http://hadoopsummit.org/san-jose/}Hadoop Summit 2013, San Jose, CA, USA}}
-
- * {{{http://www.slideshare.net/Hadoop_Summit/murhty-saha-june26255pmroom212}Slides}}
-
- * {{{http://www.youtube.com/watch?v=9ZLLzlsz7h8}Video}}
-
-User Meetup Recordings
-
- * {{{https://hortonworks.webex.com/hortonworks/ldr.php?AT=pb&SP=MC&rID=125516477&rKey=d147a3c924b64496}Recording}} from {{{http://www.meetup.com/Apache-Tez-User-Group/events/130852782/}Meetup on July 31st, 2013}} at {{{hortonworks.com}Hortonworks Inc}}
-
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/index.md b/docs/src/site/markdown/index.md
new file mode 100644
index 0000000..337d21c
--- /dev/null
+++ b/docs/src/site/markdown/index.md
@@ -0,0 +1,47 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Welcome to Apache Tez</title></head>
+
+Introduction
+------------
+
+The Apache Tez project is aimed at building an application framework
+which allows for a complex directed-acyclic-graph of tasks for processing
+data. It is currently built atop
+[Apache Hadoop YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
+
+The 2 main design themes for Tez are:
+
+- **Empowering end users by:**
+ - Expressive dataflow definition APIs
+ - Flexible Input-Processor-Output runtime model
+ - Data type agnostic
+ - Simplifying deployment
+- **Execution Performance**
+ - Performance gains over Map Reduce
+ - Optimal resource management
+ - Plan reconfiguration at runtime
+ - Dynamic physical data flow decisions
+
+By allowing projects like Apache Hive and Apache Pig to run a complex
+DAG of tasks, Tez can be used to process data, that earlier took
+multiple MR jobs, now in a single Tez job as shown below.
+
+
+
+
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/install.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/install.md b/docs/src/site/markdown/install.md
new file mode 100644
index 0000000..cb9b586
--- /dev/null
+++ b/docs/src/site/markdown/install.md
@@ -0,0 +1,109 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Install and Deployment Instructions</title></head>
+
+[Install instructions for Tez-0.5.0-SNAPSHOT - master branch](./install_0_5_0.html)
+-----------------------------------------------------------------------------------
+
+Install/Deploy Instructions for the latest Tez release [(Tez-0.4.1 src)](http://www.apache.org/dyn/closer.cgi/incubator/tez/tez-0.4.1-incubating/)
+--------------------------------------------------------------------------------------------------------------------------------------------------
+
+1. Deploy Apache Hadoop using either the 2.2.0 release or a compatible
+ 2.x version.
+ - One thing to note though when compiling Tez is that you will
+ need to change the value of the hadoop.version property in the
+ toplevel pom.xml to match the version of the hadoop branch being
+ used.
+2. Build tez using `mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true`
+ - This assumes that you have already installed JDK6 or later,
+ Maven 3 or later and Protocol Buffers (protoc compiler) 2.5 or
+ later
+ - If you prefer to run the unit tests, remove skipTests from the
+ command above.
+ - If you would like to create a tarball of the release, use `mvn
+ clean package -Dtar -DskipTests=true -Dmaven.javadoc.skip=true`
+ - If you use Eclipse IDE, you can import the projects using
+ "Import/Maven/Existing Maven Projects". Eclipse does not
+ automatically generate Java sources or include the generated
+ sources into the projects. Please build using maven as described
+ above and then use Project Properties to include
+ "target/generated-sources/java" as a source directory into the
+ "Java Build Path" for these projects: tez-api, tez-mapreduce,
+ tez-runtime-internals and tez-runtime-library. This needs to be done
+ just once after importing the project.
+3. Copy the tez jars and their dependencies into HDFS.
+ - The tez jars and dependencies will be found in
+ tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating if you run
+ the intial command mentioned in step 2.
+ - Assuming that the tez jars are put in /apps/ on HDFS, the
+ command would be `hadoop dfs -put
+ tez-dist/target/tez-0.4.1-incubating/tez-0.4.1-incubating /apps/`
+ - Please do not upload the tarball to HDFS, upload only the jars.
+4. Configure tez-site.xml to set tez.lib.uris to point to the paths in
+ HDFS containing the jars. Please note that the paths are not
+ searched recursively so for *basedir* and *basedir*/lib/, you will
+ need to configure the 2 paths as a comma-separated list. * Assuming
+ you followed step 3, the value would be:
+ "${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name}/apps/tez-0.4.1-incubating/lib/"
+5. Modify mapred-site.xml to change _mapreduce.framework.name_ property
+ from its default value of *yarn* to *yarn-tez*
+6. Set HADOOP_CLASSPATH to have the following paths in it:
+ - TEZ_CONF_DIR - location of tez-site.xml
+ - TEZ_JARS and TEZ_JARS/libs - location of the tez jars and
+ dependencies.
+ - The command to set up the classpath should be something like:
+ `export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*`
+ Please note the "*" which is an important requirement when
+ setting up classpaths for directories containing jar files.
+7. Submit a MR job as you normally would using something like:
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
+ ```
+
+ This will use the TEZ DAG ApplicationMaster to run the MR job. This
+ can be verified by looking at the AM’s logs from the YARN
+ ResourceManager UI.
+8. There is a basic example of using an MRR job in the
+ tez-mapreduce-examples.jar. Refer to OrderedWordCount.java in the
+ source code. To run this example:
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input> <output>
+ ```
+
+ This will use the TEZ DAG ApplicationMaster to run the ordered word
+ count job. This job is similar to the word count example except that
+ it also orders all words based on the frequency of occurrence.
+
+ There are multiple variations to run orderedwordcount. You can use
+ it to run multiple DAGs serially on different inputs/outputs. These
+ DAGs could be run separately as different applications or serially
+ within a single TEZ session.
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
+ ```
+
+ The above will run multiple DAGs for each input-output pair.
+
+ To use TEZ sessions, set -DUSE_TEZ_SESSION=true
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-mapreduce-examples.jar orderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
+ ```
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/install_0_5_0.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/install_0_5_0.md b/docs/src/site/markdown/install_0_5_0.md
new file mode 100644
index 0000000..2cf0e3d
--- /dev/null
+++ b/docs/src/site/markdown/install_0_5_0.md
@@ -0,0 +1,117 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Install and Deployment Instructions</title></head>
+
+Install/Deploy Instructions for Tez-current (0.5.0-SNAPSHOT, branch master)
+---------------------------------------------------------------------------
+
+1. Deploy Apache Hadoop using either the 2.2.0 release or a compatible
+ 2.x version.
+ - One thing to note though when compiling Tez is that you will
+ need to change the value of the hadoop.version property in the
+ top-level pom.xml to match the version of the hadoop branch being
+ used.
+2. Build tez using `mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true`
+ - This assumes that you have already installed JDK6 or later,
+ Maven 3 or later and Protocol Buffers (protoc compiler) 2.5 or
+ later
+ - If you prefer to run the unit tests, remove skipTests from the
+ command above.
+ - If you use Eclipse IDE, you can import the peojects using
+ "Import/Maven/Existing Maven Projects". Eclipse does not
+ automatically generate Java sources or include the generated
+ sources into the projects. Please build using maven as described
+ above and then use Project Properties to include
+ "target/generatedsources/java" as a source directory into the
+ "Java Build Path" for these projects: tez-api, tez-mapreduce,
+ tez-runtime-internals and tez-runtime-library. This needs to be done
+ just once after importing the project.
+3. Copy the relevant tez tarball into HDFS, and configure tezsite.xml
+ - A tez tarball containing tez and hadoop libraries will be found
+ at tez-dist/target/tez-0.5.0-SNAPSHOT.tar.gz
+ - Assuming that the tez jars are put in /apps/ on HDFS, the
+ command would be
+ ```
+ hadoop dfs -mkdir /apps/tez-0.5.0-SNAPSHOT
+ hadoop dfs -copyFromLocal tez-dist/target/tez-0.5.0-SNAPSHOT-archive.tar.gz /apps/tez-0.5.0-SNAPSHOT/
+ ```
+ - tez-site.xml configuration.
+ - Set tez.lib.uris to point to the tar.gz uploaded to HDFS.
+ Assuming the steps mentioned so far were followed,
+ ```
+ set tez.lib.uris to "${fs.default.name}/apps/tez-0.5.0-SNAPSHOT/tez-0.5.0-SNAPSHOT.tar.gz"
+ ```
+ - Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml,
+ or if it is set, the value should be false
+4. Optional: If running existing MapReduce jobs on Tez. Modify
+ mapred-site.xml to change "mapreduce.framework.name" property from
+ its default value of "yarn" to "yarn-tez"
+5. Configure the client node to include the tez-libraries in the hadoop
+ classpath
+ - Extract the tez tarball created in step 2 to a local directory
+ (assuming TEZ_JARS is where the files will be decompressed for
+ the next steps)
+ ```
+ tar -xvzf tez-dist/target/tez-0.5.0-SNAPSHOT.tar.gz -C $TEZ_JARS
+ ```
+ - set TEZ_CONF_DIR to the location of tez-site.xml
+ - The command to set up the classpath should be something like:
+ ```
+ export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
+ ```
+ - Please note the "*" which is an important requirement when
+ setting up classpaths for directories containing jar files.
+6. Submit a MR job as you normally would using something like:
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
+ ```
+
+ This will use the TEZ DAG ApplicationMaster to run the MR job. This
+ can be verified by looking at the AM’s logs from the YARN
+ ResourceManager UI.
+
+7. There is a basic example of using an MRR job in the tez-examples.jar.
+ Refer to OrderedWordCount.java in the source code. To run this
+ example:
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount <input> <output>
+ ```
+
+ This will use the TEZ DAG ApplicationMaster to run the ordered word
+ count job. This job is similar to the word count example except that
+ it also orders all words based on the frequency of occurrence.
+
+ Tez DAGs could be run separately as different applications or
+ serially within a single TEZ session. There is a different variation
+ of orderedwordcount in tez-tests that supports the use of Sessions
+ and handling multiple input-output pairs. You can use it to run
+ multiple DAGs serially on different inputs/outputs.
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
+ ```
+
+ The above will run multiple DAGs for each input-output pair.
+
+ To use TEZ sessions, set -DUSE_TEZ_SESSION=true
+
+ ```
+ $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
+ ```
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/localmode.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/localmode.md b/docs/src/site/markdown/localmode.md
new file mode 100644
index 0000000..ae546a1
--- /dev/null
+++ b/docs/src/site/markdown/localmode.md
@@ -0,0 +1,107 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Tez Local Mode</title></head>
+
+Tez Local Mode
+--------------
+
+Tez Local Mode is a development tool to test Tez jobs without needing to
+bring up a Hadoop cluster. Local Mode runs the Tez components
+AppMaster, TaskRunner that are used when executing a job on a cluster.
+From a developer tool perspective, it offers several advantages.
+
+- Fast prototyping Hadoop setup, launch cost etc not involved.
+- Unit testing Fast execution since the overhead of allocating
+ resources, launching JVMs etc is removed.
+- Easy debuggability Single JVM running all user code.
+
+While majority of the components are reused in Local Mode, there are
+some bits which are not
+
+- Scheduling and Container Re-Use differs.
+- Handling of YARN Local Resources. Local Mode expects necessary jars
+ to be loaded with the Client when executing.
+- Contains some performance improvements like skipping RPC invocations
+ since everything runs within the same JVM.
+
+Running a DAG in Local Mode
+
+- "tez.local.mode" should be set to true in the confgiuration instance
+ used to create the TezClient.
+- The FileSystem must be configured to the local file system
+ ("fs.default.name" must be set to "file<span></span>:///"). This is required to be
+ setup in all Configuration instances used to create a DAG.
+ Typically, when using Tez for testing and prototyping without a
+ Hadoop cluster, this is not a problem. It becomes a problem when
+ Hadoop Configuration files are in the classpath, with a different
+ default filesystem configured.
+- Setup the fetchers to make use of local reads instead of fetching
+ from remote nodes. ("tez.runtime.optimize.local.fetch" must be set to true)
+- Beyond this, no other changes are required, to make use of Local
+ Mode instead of running a job on a cluster.
+- If using this in code, the following changes should be made to
+ configuration, after which this configuration instance becomes the
+ base for all other Configuration instances.
+
+ ```
+ Configuration conf = new Configuration();
+ conf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
+ conf.set("fs.default.name", "file:///");
+ conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH, true);
+ ```
+
+- If using a tez-site.xml config file, it should contain the following
+ entries
+
+ ```
+ <property>
+ <name>fs.default.name</name>
+ <value>file:///</value>
+ </property>
+ <property>
+ <name>tez.local.mode</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>tez.runtime.optimize.local.fetch</name>
+ <value>true</value>
+ </property>
+ ```
+
+Things to watch out for
+
+- In current Local Mode, large amount of input data may lead to JVM
+ out of memory since all TEZ components are running in single JVM.
+ The input data size should be kept small.
+- TezConfiguration.TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS(tez.am.inline.task.execution.maxtasks)
+ should not be changed (defaults to 1).
+- "tez.history.logging.service.class" should be the default value:
+ "org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService".
+ It means ATS is disabled in current Local Mode.
+
+Potential pitfalls when moving from Local Mode to a real cluster
+
+- Resource requirements (CPU, Memory, etc) which would otherwise have
+ been specified for a YARN Cluster will now start taking affect, and
+ should be considered.
+- The Java Options and Environment variables which may have been setup
+ for the DAG do not take affect in Local Mode, and could be a source
+ of migration problems.
+- The ObjectRegistry will work within a single task, when running in
+ Local Mode. The behaviour would be different on a real cluster,
+ where it would work across tasks which share the same container.
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/privacy-policy.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/privacy-policy.md b/docs/src/site/markdown/privacy-policy.md
new file mode 100644
index 0000000..95825d0
--- /dev/null
+++ b/docs/src/site/markdown/privacy-policy.md
@@ -0,0 +1,52 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Privacy Policy</title></head>
+
+Privacy Policy
+--------------
+
+Information about your use of this website is collected using server
+access logs and a tracking cookie. The collected information consists of
+the following:
+
+1. The IP address from which you access the website;
+
+2. The type of browser and operating system you use to access our site;
+
+3. The date and time you access our site;
+
+4. The pages you visit; and
+
+5. The addresses of pages from where you followed a link to our site.
+
+Part of this information is gathered using a tracking cookie set by the
+[Google Analytics](http://www.google.com/analytics/) service and handled
+by Google as described in their [privacy
+policy](http://www.google.com/privacy.html). See your browser
+documentation for instructions on how to disable the cookie if you
+prefer not to share this data with Google.
+
+We use the gathered information to help us make our site more useful to
+visitors and to better understand how and when our site is used. We do
+not track or collect personally identifiable information or associate
+gathered data with any personally identifying information from other
+sources.
+
+By using this website, you consent to the collection of this data in the
+manner and for the purpose described above.
+
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/docs/src/site/markdown/talks.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/talks.md b/docs/src/site/markdown/talks.md
new file mode 100644
index 0000000..8971088
--- /dev/null
+++ b/docs/src/site/markdown/talks.md
@@ -0,0 +1,32 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<head><title>Presentations and Talks on Tez</title></head>
+
+Talks
+-----
+- Apache Tez : Accelerating Hadoop Query Processing by Arun Murthy and
+ Bikas Saha at [Hadoop Summit 2013, San Jose, CA, USA](http://hadoopsummit.org/san-jose/)
+ - [Slides](http://www.slideshare.net/Hadoop_Summit/murhty-saha-june26255pmroom212)
+ - [Video](http://www.youtube.com/watch?v=9ZLLzlsz7h8)
+
+User Meetup Recordings
+----------------------
+
+- [Recording](https://hortonworks.webex.com/hortonworks/ldr.php?AT=pb&SP=MC&rID=125516477&rKey=d147a3c924b64496)
+ from [Meetup on July 31st, 2013](http://www.meetup.com/Apache-Tez-User-Group/events/130852782/)
+ at [Hortonworks Inc](http://hortonworks.com)
http://git-wip-us.apache.org/repos/asf/tez/blob/119d5c79/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 7477c47..e090e8d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -718,7 +718,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
- <version>3.3</version>
+ <version>3.4</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>