You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by sj...@apache.org on 2015/12/11 18:54:10 UTC
[1/2] hadoop git commit: YARN-4341. add doc about timeline
performance tool usage (Chang Li via sjlee)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 5595b912e -> 605914caa
YARN-4341. add doc about timeline performance tool usage (Chang Li via sjlee)
(cherry picked from commit e8964589de27a507c31203a87c3cbe0a8d68d3dd)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8be66720
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8be66720
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8be66720
Branch: refs/heads/branch-2
Commit: 8be667208f73bfc36515394b0aee809069dc2a3d
Parents: 5595b91
Author: Sangjin Lee <sj...@apache.org>
Authored: Fri Dec 11 09:46:32 2015 -0800
Committer: Sangjin Lee <sj...@apache.org>
Committed: Fri Dec 11 09:53:17 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +
.../src/site/markdown/TimelineServer.md | 78 ++++++++++++++++++++
2 files changed, 81 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8be66720/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 73eb4ec..6599f9e 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -555,6 +555,9 @@ Release 2.8.0 - UNRELEASED
YARN-4248. REST API for submit/update/delete Reservations. (curino)
+ YARN-4341. add doc about timeline performance tool usage (Chang Li via
+ sjlee)
+
OPTIMIZATIONS
YARN-3339. TestDockerContainerExecutor should pull a single image and not
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8be66720/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
index 2048012..3e589d3 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
@@ -26,6 +26,10 @@ The YARN Timeline Server
* [Publishing of application specific data](#Publishing_of_application_specific_data)
* [Timeline Server REST API](#Timeline_Server_REST_API_v1)
* [Generic Data REST APIs](#GENERIC_DATA_REST_APIS)
+* [Timelnine Server Performance Test Tool](#TIMELINE_SERVER_PERFORMANCE_TEST_TOOL)
+ * [Highlights](#HIGHLIGHTS)
+ * [Usage](#USAGE)
+ * [Sample Runs](#SAMPLE_RUNS)
<a name="Overview"></a>Overview
---------
@@ -2033,3 +2037,77 @@ This hides details of other domains from an unauthorized caller.
this failure *will not* result in an HTTP error code being retured.
A status code of 200 will be returned —however, there will be an error code
in the list of failed entities for each entity which could not be added.
+
+<a name="TIMELINE_SERVER_PERFORMANCE_TEST_TOOL"></a> Timelnine Server Performance Test Tool
+----------
+###<a name="HIGHLIGHTS"></a>Highlights
+
+The timeline server performance test tool helps measure timeline server's write performance. The test
+launches SimpleEntityWriter mappers or JobHistoryFileReplay mappers to write timeline
+entities to the timeline server. At the end, the transaction rate(ops/s) per mapper and the total transaction rate
+will be measured and printed out. Running the test with SimpleEntityWriter mappers
+will also measure and show the IO rate(KB/s) per mapper and the total IO rate.
+
+###<a name="USAGE"></a>Usage
+
+Mapper Types Description:
+
+ 1. SimpleEntityWriter mapper
+ Each mapper writes a user-specified number of timeline entities
+ with a user-specified size to the timeline server.
+
+ 2. JobHistoryFileReplay mapper
+ Each mapper replays jobhistory files under a specified directory
+ (both the jhist file and its corresponding conf.xml are required to
+ be present in order to be replayed. The number of mappers should be no more
+ than the number of jobhistory files).
+ Each mapper will get assigned some jobhistory files to replay. For each
+ job history file, a mapper will parse it to get jobinfo and then create
+ timeline entities. Each mapper also has the choice to write all the
+ timeline entities created at once or one at a time.
+
+Options:
+
+ [-m <maps>] number of mappers (default: 1)
+ [-v] timeline service version
+ [-mtype <mapper type in integer>]
+ 1. simple entity write mapper
+ 2. jobhistory files replay mapper
+ [-s <(KBs)test>] number of KB per put (mtype=1, default: 1 KB)
+ [-t] package sending iterations per mapper (mtype=1, default: 100)
+ [-d <path>] root path of job history files (mtype=2)
+ [-r <replay mode>] (mtype=2)
+ 1. write all entities for a job in one put (default)
+ 2. write one entity at a time
+
+###<a name="SAMPLE_RUNS"></a>Sample Runs
+
+Run SimpleEntityWriter test:
+
+ bin/hadoop jar performanceTest.jar timelineperformance -m 4 -mtype 1 -s 3 -t 200
+
+Example output of SimpleEntityWriter test :
+
+ TRANSACTION RATE (per mapper): 20000.0 ops/s
+ IO RATE (per mapper): 60000.0 KB/s
+ TRANSACTION RATE (total): 80000.0 ops/s
+ IO RATE (total): 240000.0 KB/s
+
+Run JobHistoryFileReplay mapper test
+
+ $ bin/hadoop jar performanceTest.jar timelineperformance -m 2 -mtype 2 -d /testInput -r 2
+
+Example input of JobHistoryFileReplay mapper test:
+
+ $ bin/hadoop fs -ls /testInput
+ /testInput/job_1.jhist
+ /testInput/job_1_conf.xml
+ /testInput/job_2.jhist
+ /testInput/job_2_conf.xml
+
+Eample output of JobHistoryFileReplay test:
+
+ TRANSACTION RATE (per mapper): 4000.0 ops/s
+ IO RATE (per mapper): 0.0 KB/s
+ TRANSACTION RATE (total): 8000.0 ops/s
+ IO RATE (total): 0.0 KB/s
[2/2] hadoop git commit: YARN-4341 addendum. Moved the changelog
entry from 2.8.0 to 2.9.0.
Posted by sj...@apache.org.
YARN-4341 addendum. Moved the changelog entry from 2.8.0 to 2.9.0.
(cherry picked from commit b7b292367ff611f5025605304db9f24a6cfe5653)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/605914ca
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/605914ca
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/605914ca
Branch: refs/heads/branch-2
Commit: 605914caac80c73f2c847a858c861d968c6da029
Parents: 8be6672
Author: Sangjin Lee <sj...@apache.org>
Authored: Fri Dec 11 09:51:24 2015 -0800
Committer: Sangjin Lee <sj...@apache.org>
Committed: Fri Dec 11 09:53:40 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/605914ca/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 6599f9e..53659c6 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -12,6 +12,9 @@ Release 2.9.0 - UNRELEASED
ResourceManager to support JvmPauseMonitor as a service.
(Sunil G via Stevel)
+ YARN-4341. add doc about timeline performance tool usage (Chang Li via
+ sjlee)
+
OPTIMIZATIONS
BUG FIXES
@@ -555,9 +558,6 @@ Release 2.8.0 - UNRELEASED
YARN-4248. REST API for submit/update/delete Reservations. (curino)
- YARN-4341. add doc about timeline performance tool usage (Chang Li via
- sjlee)
-
OPTIMIZATIONS
YARN-3339. TestDockerContainerExecutor should pull a single image and not