You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ja...@apache.org on 2017/09/13 11:24:56 UTC
carbondata git commit: [CARBONDATA-1117]updated
configuration-parameters.md and useful-tips-on-carbondata.md files for
SET/RESET
Repository: carbondata
Updated Branches:
refs/heads/master 2176a2f1d -> b414393be
[CARBONDATA-1117]updated configuration-parameters.md and useful-tips-on-carbondata.md files for SET/RESET
This closes #1219
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/b414393b
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/b414393b
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/b414393b
Branch: refs/heads/master
Commit: b414393bed319ea970036aac72f140b9389bccf9
Parents: 2176a2f
Author: vandana <va...@gmail.com>
Authored: Mon Jul 31 19:32:58 2017 +0530
Committer: Jacky Li <ja...@qq.com>
Committed: Wed Sep 13 19:24:35 2017 +0800
----------------------------------------------------------------------
docs/configuration-parameters.md | 87 +++++++++++++++++++++++++++++++++-
docs/useful-tips-on-carbondata.md | 1 -
2 files changed, 85 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata/blob/b414393b/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index bdd551a..8101aa1 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -24,6 +24,7 @@
* [Performance Configuration](#performance-configuration)
* [Miscellaneous Configuration](#miscellaneous-configuration)
* [Spark Configuration](#spark-configuration)
+ * [Dynamic Configuration In CarbonData Using SET-RESET](#dynamic-configuration-in-carbondata-using-set-reset)
## System Configuration
@@ -146,5 +147,87 @@ This section provides the details of all the configurations required for CarbonD
|----------------------------------------|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| spark.driver.memory | 1g | Amount of memory to be used by the driver process. |
| spark.executor.memory | 1g | Amount of memory to be used per executor process. |
-
-
+
+## Dynamic Configuration In CarbonData Using SET-RESET
+
+**SET/RESET** commands are used to add, update, display, or reset the carbondata properties dynamically without restarting the driver.
+
+**Syntax**
+
+* **Add or Update :** This command adds or updates the value of parameter_name.
+
+```
+SET parameter_name=parameter_value
+```
+
+* Display Property Value: This command displays the value of the specified parameter_name.
+
+```
+SET parameter_name
+```
+
+* Display Session Parameters: This command displays all the supported session parameters.
+
+```
+SET
+```
+
+* Display Session Parameters along with usage details: This command displays all the supported session parameters along with their usage details.
+
+```
+SET -v
+```
+
+* Reset: This command clears all the session parameters.
+
+```
+RESET
+```
+
+ **Parameter Description:**
+
+| Parameter | Description |
+|-----------------|----------------------------------------------------------------------------------------|
+| parameter_name | Name of the property whose value needs to be dynamically added, updated, or displayed. |
+| parameter_value | New value of the parameter_name to be set. |
+
+<b><p align="center">Dynamically Configurable Properties of CarbonData</p></b>
+
+| Properties | Description |
+|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| carbon.options.bad.records.logger.enable | To enable or disable bad record logger. |
+| carbon.options.bad.records.action | This property can have four types of actions for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found. |
+| carbon.options.is.empty.data.bad.record | If false, then empty ("" or '' or ,,) data will not be considered as bad record and vice versa. |
+| carbon.options.sort.scope | This property can have four possible values BATCH_SORT, LOCAL_SORT, GLOBAL_SORT and NO_SORT. If set to BATCH_SORT, the sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. If set to LOCAL_SORT, the sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. If set to GLOBAL_SORT, the sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. If set to NO_SORT data will be loaded in unsorted manner. |
+| carbon.options.batch.sort.size.inmb | Size of batch data to keep in memory, as a thumb rule it supposed to be less than 45% of sort.inmemory.size.inmb otherwise it may spill intermediate data to disk. |
+| carbon.options.single.pass | Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. This option specifies whether to use single pass for loading data or not. By default this option is set to FALSE. |
+| carbon.options.bad.record.path | Specifies the HDFS path where bad records needs to be stored. |
+| carbon.options.global.sort.partitions | The Number of partitions to use when shuffling data for sort. If user don't configurate or configurate it less than 1, it uses the number of map tasks as reduce tasks. In general, we recommend 2-3 tasks per CPU core in your cluster. |
+| carbon.custom.block.distribution | Specifies whether to use the Spark or Carbon block distribution feature. |
+| enable.unsafe.sort | Specifies whether to use unsafe sort during data loading. Unsafe sort reduces the garbage collection during data load operation, resulting in better performance. |
+
+**Examples:**
+
+* Add or Update:
+
+```
+SET enable.unsafe.sort =true
+```
+
+* Display Property Value:
+
+```
+SET enable.unsafe.sort
+```
+
+* Reset:
+
+```
+RESET
+```
+
+**System Response:**
+
+* Success will be recorded in the driver log.
+
+* Failure will be displayed in the UI.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata/blob/b414393b/docs/useful-tips-on-carbondata.md
----------------------------------------------------------------------
diff --git a/docs/useful-tips-on-carbondata.md b/docs/useful-tips-on-carbondata.md
index 6c73b5e..d1d4a8c 100644
--- a/docs/useful-tips-on-carbondata.md
+++ b/docs/useful-tips-on-carbondata.md
@@ -25,7 +25,6 @@ The following sections will elaborate on the above topics :
* [Configuration for Optimizing Data Loading performance for Massive Data](#configuration-for-optimizing-data-loading-performance-for-massive-data)
* [Optimizing Mass Data Loading](#configurations-for-optimizing-carbondata-performance)
-
## Suggestions to Create CarbonData Table
Recently CarbonData was used to analyze performance of Telecommunication field.