You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tajo.apache.org by hy...@apache.org on 2013/12/07 20:43:49 UTC
git commit: Update documentation by TAJO-317. (Keuntae Park, jihoon,
and hyunsik)
Updated Branches:
refs/heads/master 3c2a6343f -> 7e47f6b01
Update documentation by TAJO-317. (Keuntae Park, jihoon, and hyunsik)
Project: http://git-wip-us.apache.org/repos/asf/incubator-tajo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tajo/commit/7e47f6b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tajo/tree/7e47f6b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tajo/diff/7e47f6b0
Branch: refs/heads/master
Commit: 7e47f6b017717fcc8917b6881fc90c4ade37a8b2
Parents: 3c2a634
Author: Hyunsik Choi <hy...@apache.org>
Authored: Sun Dec 8 04:42:37 2013 +0900
Committer: Hyunsik Choi <hy...@apache.org>
Committed: Sun Dec 8 04:42:37 2013 +0900
----------------------------------------------------------------------
.../src/site/markdown/tajo-0.8.0-doc.md | 60 ++++++++++++++++++--
1 file changed, 56 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tajo/blob/7e47f6b0/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
----------------------------------------------------------------------
diff --git a/tajo-project/src/site/markdown/tajo-0.8.0-doc.md b/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
index 2de57f8..17aaa67 100644
--- a/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
+++ b/tajo-project/src/site/markdown/tajo-0.8.0-doc.md
@@ -32,6 +32,7 @@
* [Launch a Tajo Cluster](#LaunchATajoCluster)
* [First Query Execution](#FirstQueryExecution)
* [Distributed mode on HDFS cluster](#DistributedMode)
+ * [Note] (#TutorialNote)
* [Configuration](#Configuration)
* [Preliminary](#Preliminary)
* [catalog-site.xml and tajo-site.xml](#catalog-site_and_tajo-site)
@@ -42,6 +43,7 @@
* [TajoMaster Heap Memory Size](#TajoMasterHeap)
* [Temporary Data Directory](#TemporaryDataDir)
* [Maximum number of parallel running tasks for each worker](#MaximumParallelRunningTasks)
+ * [Worker Resource] (#ResourceConfiguration)
* [Catalog Configuration](#CatalogConfiguration)
* [RPC/Http Service Configuration and Default Addresses](#DefaultPorts)
* [Tajo Master](#TajoMasterDefaultPorts)
@@ -249,6 +251,11 @@ Then, execute start-tajo.sh
$ $TAJO_HOME/bin/start-tajo.sh
```
+## <a name="TutorialNote"></a>Note
+ * In default, each worker is set to very little resource capacity. In order to increase parallel degree, see [this section] (#MaximumParallelRunningTasks).
+ * In default, TajoMaster listens on 127.0.0.1 for clients. To allow remote clients to access TajoMaster, please set **tajo.master.client-rpc.address** config to tajo-site.xml. You can get more information from [RPC/Http Service Configuration and Default Addresses](#DefaultPorts) section.
+
+
Enjoy Apache Tajo!
# <a name="Configuration"></a>Configuration
@@ -342,19 +349,64 @@ TajoWorker stores temporary data on local file system due to out-of-core algorit
### <a name="MaximumParallelRunningTasks"></a>Maximum number of parallel running tasks for each worker
-Each worker can execute multiple tasks at a time. Tajo allows users to specify the maximum number of parallel running tasks for each worker.
+In Tajo, the capacity of running tasks in parallel are determined by available resources and workload of running queries. In order to specify it, please see [Worker Resources] (#ResourceConfiguration) section.
+
+### <a name="ResourceConfiguration"></a>Worker Resources
+
+Each worker can execute multiple tasks simultaneously.
+In Tajo, users can specify the total size of memory and the number of disks for each worker. Available resources affect how many tasks are executed simultaneously.
+
+In order to specify the resource capacity of each worker, you should add the following configs to *tajo-site.xml* :
+
+| property name | description | value type | default value |
+| ------------------------------- | ------------------------ | ------------------- | ----------------------- |
+| tajo.worker.resource.cpu-cores | the number of cpu cores | integer | 1 |
+| tajo.worker.resource.memory-mb | memory size (MB) | integer | 1024 |
+| tajo.worker.resource.disks | the number of disks | integer | 1 |
+
+_NOTE: Currently, QueryMaster requests 512MB memory and 1.0 disk per task for the backward compatibility._
+
+#### Example
+
+Assume that you want to give 5120 MB memory, 6.0 disks, and 24 cores on each worker. The example configuration is as follows:
*tajo-site.xml*
```
<property>
- <name>tajo.worker.parallel-execution.max-num</name>
- <value>12</value>
+ <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
+ <value>24</value>
+ </property>
+
+ <property>
+ <name>tajo.worker.resource.memory-mb</name>
+ <value>5120</value>
</property>
+
+ <property>
+ <name>tajo.worker.resource.tajo.worker.resource.disks</name>
+ <value>6.0</value>
+ </property>
```
+#### Dedicated Mode
+Tajo provides a dedicated mode that allows each worker in a Tajo cluster to use whole available system resources including cpu-cores, memory, and disks. For this mode, a user should add the following config to *tajo-site.xml* :
+
+```xml
+ <property>
+ <name>tajo.worker.resource.dedicated</name>
+ <value>true</value>
+ </property>
+```
+
+In addition, it can limit the memory capacity used for Tajo worker as follows:
+
+| property name | description | value type | default value |
+| ---------------------------------------------| ------------------------------------------------ | ------------------- | ----------------------- |
+| tajo.worker.resource.dedicated-memory-ratio | how much memory to be used in whole memory | float | 0.8 |
+
## <a name="CatalogConfiguration"></a>Catalog Configuration
-If you want to customize the catalog service, copy $TAJO_HOME/conf/catalog-site.xml.templete to catalog-site.xml. Then, add the following configs to catalog-site.xml. Note that the default configs are enough to launch Tajo cluster in most cases.
+If you want to customize the catalog service, copy $TAJO_HOME/conf/catalog-site.xml.template to catalog-site.xml. Then, add the following configs to catalog-site.xml. Note that the default configs are enough to launch Tajo cluster in most cases.
* tajo.catalog.master.addr - If you want to launch a Tajo cluster in distributed mode, you must specify this address. For more detail information, see [Default Ports](#DefaultPorts).
* tajo.catalog.store.class - If you want to change the persistent storage of the catalog server, specify the class name. Its default value is tajo.catalog.store.DerbyStore. In the current version, Tajo provides three persistent storage classes as follows: