You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by xu...@apache.org on 2018/12/14 11:52:45 UTC
carbondata git commit: [CARBONDATA-3166]Updated Document and added
Column Compressor used in Describe Formatted
Repository: carbondata
Updated Branches:
refs/heads/master 82adc50e7 -> ebdd5486e
[CARBONDATA-3166]Updated Document and added Column Compressor used in Describe Formatted
Updated Document and added column compressor used in Describe Formatted Command
This closes #2986
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/ebdd5486
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/ebdd5486
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/ebdd5486
Branch: refs/heads/master
Commit: ebdd5486e309e606efa1ddeb7d2b6d62f2315541
Parents: 82adc50
Author: shardul-cr7 <sh...@gmail.com>
Authored: Thu Dec 13 14:12:18 2018 +0530
Committer: xuchuanyin <xu...@hust.edu.cn>
Committed: Fri Dec 14 19:52:34 2018 +0800
----------------------------------------------------------------------
docs/configuration-parameters.md | 2 +-
.../execution/command/table/CarbonDescribeFormattedCommand.scala | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ebdd5486/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index 4aa2929..db21c6a 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -91,7 +91,7 @@ This section provides the details of all the configurations required for the Car
| carbon.dictionary.server.port | 2030 | Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. Single pass loading can be enabled using the option ***carbon.options.single.pass***. When this option is specified, a dictionary server will be internally started to handle the dictionary generation and query requests. This configuration specifies the port on which the server need to listen for incoming requests. Port value ranges between 0-65535 |
| carbon.load.directWriteToStorePath.enabled | false | During data load, all the carbondata files are written to local disk and finally copied to the target store location in HDFS/S3. Enabling this parameter will make carbondata files to be written directly onto target HDFS/S3 location bypassing the local disk.**NOTE:** Writing directly to HDFS/S3 saves local disk IO(once for writing the files and again for copying to HDFS/S3) there by improving the performance. But the drawback is when data loading fails or the application crashes, unwanted carbondata files will remain in the target HDFS/S3 location until it is cleared during next data load or by running *CLEAN FILES* DDL command |
| carbon.options.serialization.null.format | \N | Based on the business scenarios, some columns might need to be loaded with null values. As null value cannot be written in csv files, some special characters might be adopted to specify null values. This configuration can be used to specify the null values format in the data being loaded. |
-| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy' and 'zstd' compressors. |
+| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy', 'zstd' and 'gzip' compressors. |
| carbon.minmax.allowed.byte.count | 200 | CarbonData will write the min max values for string/varchar types column using the byte count specified by this configuration. Max value is 1000 bytes(500 characters) and Min value is 10 bytes(5 characters). **NOTE:** This property is useful for reducing the store size thereby improving the query performance but can lead to query degradation if value is not configured properly. | |
## Compaction Configuration
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ebdd5486/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
----------------------------------------------------------------------
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
index 151359e..2d560df 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
@@ -92,7 +92,9 @@ private[sql] case class CarbonDescribeFormattedCommand(
Strings.formatSize(
tblProps.getOrElse(CarbonCommonConstants.CARBON_LOAD_MIN_SIZE_INMB,
CarbonCommonConstants.CARBON_LOAD_MIN_SIZE_INMB_DEFAULT).toFloat), ""),
-
+ ("Data File Compressor ", tblProps
+ .getOrElse(CarbonCommonConstants.COMPRESSOR,
+ CarbonCommonConstants.DEFAULT_COMPRESSOR), ""),
//////////////////////////////////////////////////////////////////////////////
// Index Information
//////////////////////////////////////////////////////////////////////////////