You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ja...@apache.org on 2017/10/01 12:43:23 UTC
carbondata git commit: [CARBONDATA-1536] Default value of
carbon.bad.records.action is FORCE
Repository: carbondata
Updated Branches:
refs/heads/master 349c59c7b -> 7d3bf0503
[CARBONDATA-1536] Default value of carbon.bad.records.action is FORCE
Default value of carbon.bad.records.action is 'FORCE' in source code, but the value is 'FAIL' in documentation. This PR fix documentation
This closes #1396
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/7d3bf050
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/7d3bf050
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/7d3bf050
Branch: refs/heads/master
Commit: 7d3bf05030e4f3112aa487438307d3e93bf950c5
Parents: 349c59c
Author: xubo245 <60...@qq.com>
Authored: Thu Sep 28 23:15:21 2017 +0800
Committer: Jacky Li <ja...@qq.com>
Committed: Sun Oct 1 20:43:04 2017 +0800
----------------------------------------------------------------------
docs/configuration-parameters.md | 2 +-
docs/dml-operation-on-carbondata.md | 2 +-
.../spark/testsuite/dataload/TestBatchSortDataLoad.scala | 6 +-----
3 files changed, 3 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index 77320f6..e085317 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -60,7 +60,7 @@ This section provides the details of all the configurations required for CarbonD
| carbon.update.persist.enable | true | Enabling this parameter considers persistent data. Enabling this will reduce the execution time of UPDATE operation. | |
| carbon.load.global.sort.partitions | 0 | The Number of partitions to use when shuffling data for sort. If user don't configurate or configurate it less than 1, it uses the number of map tasks as reduce tasks. In general, we recommend 2-3 tasks per CPU core in your cluster.
| carbon.options.bad.records.logger.enable | false | Whether to create logs with details about bad records. | |
-| carbon.bad.records.action | fail | This property can have four types of actions for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found. | |
+| carbon.bad.records.action | FORCE | This property can have four types of actions for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found. | |
| carbon.options.is.empty.data.bad.record | false | If false, then empty ("" or '' or ,,) data will not be considered as bad record and vice versa. | |
| carbon.options.bad.record.path | | Specifies the HDFS path where bad records are stored. By default the value is Null. This path must to be configured by the user if bad record logger is enabled or bad record action redirect. | |
| carbon.enable.vector.reader | true | This parameter increases the performance of select queries as it fetch columnar batch of size 4*1024 rows instead of fetching data row by row. | |
http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/docs/dml-operation-on-carbondata.md
----------------------------------------------------------------------
diff --git a/docs/dml-operation-on-carbondata.md b/docs/dml-operation-on-carbondata.md
index 794bbc2..66109e8 100644
--- a/docs/dml-operation-on-carbondata.md
+++ b/docs/dml-operation-on-carbondata.md
@@ -198,7 +198,7 @@ OPTIONS('BAD_RECORDS_LOGGER_ENABLE'='true',
| Options | Default Value | Description |
|---------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| BAD_RECORDS_LOGGER_ENABLE | false | Whether to create logs with details about bad records. |
- | BAD_RECORDS_ACTION | FAIL | Following are the four types of action for bad records: FORCE: Auto-corrects the data by storing the bad records as NULL. REDIRECT: Bad records are written to the raw CSV instead of being loaded. IGNORE: Bad records are neither loaded nor written to the raw CSV. FAIL: Data loading fails if any bad records are found. NOTE: In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and the load operation fails. |
+ | BAD_RECORDS_ACTION | FORCE | Following are the four types of action for bad records: FORCE: Auto-corrects the data by storing the bad records as NULL. REDIRECT: Bad records are written to the raw CSV instead of being loaded. IGNORE: Bad records are neither loaded nor written to the raw CSV. FAIL: Data loading fails if any bad records are found. NOTE: In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and the load operation fails. |
| IS_EMPTY_DATA_BAD_RECORD | false | If false, then empty ("" or '' or ,,) data will not be considered as bad record and vice versa. |
| BAD_RECORD_PATH | - | Specifies the HDFS path where bad records are stored. By default the value is Null. This path must to be configured by the user if bad record logger is enabled or bad record action redirect. |
http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
----------------------------------------------------------------------
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
index 46a4f8c..4c6c56d 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
@@ -61,15 +61,11 @@ class TestBatchSortDataLoad extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS carbon_load6")
}
-
-
override def beforeAll {
dropTable
buildTestData
}
-
-
-
+
test("test batch sort load by passing option to load command") {
sql(