You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ch...@apache.org on 2018/03/13 15:27:45 UTC
[1/3] carbondata-site git commit: added documentation for release
1.3.1
Repository: carbondata-site
Updated Branches:
refs/heads/asf-site e4ea53ff1 -> 7aad1de9c
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/preaggregate-datamap-guide.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/preaggregate-datamap-guide.md b/src/site/markdown/preaggregate-datamap-guide.md
index 199f674..84f06c4 100644
--- a/src/site/markdown/preaggregate-datamap-guide.md
+++ b/src/site/markdown/preaggregate-datamap-guide.md
@@ -181,7 +181,7 @@ SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
SELECT sex, sum(quantity) from sales GROUP BY sex
-SELECT sum(price), country from sales GROUP BY country
+SELECT avg(price), country from sales GROUP BY country
```
will be transformed by CarbonData's query planner to query against pre-aggregate table
@@ -211,7 +211,7 @@ tables are loaded successfully, if one of these loads fails, new data are not vi
as if the load operation is not happened.
## Querying data
-As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+As a technique for query acceleration, Pre-aggregate tables cannot be queried directly.
Queries are to be made on main table. While doing query planning, internally CarbonData will check
associated pre-aggregate tables with the main table, and do query plan transformation accordingly.
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/useful-tips-on-carbondata.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/useful-tips-on-carbondata.md b/src/site/markdown/useful-tips-on-carbondata.md
index 4d43003..ff339d0 100644
--- a/src/site/markdown/useful-tips-on-carbondata.md
+++ b/src/site/markdown/useful-tips-on-carbondata.md
@@ -169,5 +169,6 @@
| carbon.use.local.dir | spark/carbonlib/carbon.properties | Data loading | Whether use YARN local directories for multi-table load disk load balance | If this is set it to true CarbonData will use YARN local directories for multi-table load disk load balance, that will improve the data load performance. |
| carbon.use.multiple.temp.dir | spark/carbonlib/carbon.properties | Data loading | Whether to use multiple YARN local directories during table data loading for disk load balance | After enabling 'carbon.use.local.dir', if this is set to true, CarbonData will use all YARN local directories during data load for disk load balance, that will improve the data load performance. Please enable this property when you encounter disk hotspot problem during data loading. |
| carbon.sort.temp.compressor | spark/carbonlib/carbon.properties | Data loading | Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading. | The optional values are 'SNAPPY','GZIP','BZIP2','LZ4' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck. |
+ | carbon.load.skewedDataOptimization.enabled | spark/carbonlib/carbon.properties | Data loading | Whether to enable size based block allocation strategy for data loading. | When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB~1GB. |
Note: If your CarbonData instance is provided only for query, you may specify the property 'spark.speculation=true' which is in conf directory of spark.
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/pdf.xml
----------------------------------------------------------------------
diff --git a/src/site/pdf.xml b/src/site/pdf.xml
index 2232a3e..f080de2 100644
--- a/src/site/pdf.xml
+++ b/src/site/pdf.xml
@@ -28,7 +28,7 @@
<companyLogo>../../src/site/projectLogo/ApacheLogo.png</companyLogo>
<projectLogo>../../src/site/projectLogo/CarbonDataLogo.png</projectLogo>
<coverTitle>Apache CarbonData</coverTitle>
- <coverSubTitle>Ver 1.3.0 </coverSubTitle>
+ <coverSubTitle>Ver 1.3.1 </coverSubTitle>
<coverType>Documentation</coverType>
<projectName>Apache CarbonData</projectName>
<companyName>The Apache Software Foundation</companyName>
[3/3] carbondata-site git commit: add 1.3.1 release
Posted by ch...@apache.org.
add 1.3.1 release
Project: http://git-wip-us.apache.org/repos/asf/carbondata-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata-site/commit/7aad1de9
Tree: http://git-wip-us.apache.org/repos/asf/carbondata-site/tree/7aad1de9
Diff: http://git-wip-us.apache.org/repos/asf/carbondata-site/diff/7aad1de9
Branch: refs/heads/asf-site
Commit: 7aad1de9c986056e75241946fd1c975289ef6e28
Parents: e4ea53f 2e8ae8b
Author: chenliang613 <ch...@huawei.com>
Authored: Tue Mar 13 23:26:50 2018 +0800
Committer: chenliang613 <ch...@huawei.com>
Committed: Tue Mar 13 23:26:50 2018 +0800
----------------------------------------------------------------------
content/configuration-parameters.html | 45 ++++++++-----------
content/data-management-on-carbondata.html | 15 +++++++
content/faq.html | 31 +++++++------
content/file-structure-of-carbondata.html | 3 ++
content/index.html | 12 ++++-
content/installation-guide.html | 9 ++--
content/mainpage.html | 3 ++
content/pdf/CarbonData Documentation.pdf | Bin 239761 -> 240561 bytes
content/preaggregate-datamap-guide.html | 7 ++-
content/quick-start-guide.html | 3 ++
content/security.html | 3 ++
content/streaming-guide.html | 3 ++
content/supported-data-types-in-carbondata.html | 3 ++
content/timeseries-datamap-guide.html | 3 ++
content/troubleshooting.html | 3 ++
content/useful-tips-on-carbondata.html | 10 +++++
content/videogallery.html | 4 +-
src/main/scala/html/header.html | 3 ++
src/main/webapp/configuration-parameters.html | 45 ++++++++-----------
.../webapp/data-management-on-carbondata.html | 15 +++++++
src/main/webapp/faq.html | 31 +++++++------
.../webapp/file-structure-of-carbondata.html | 3 ++
src/main/webapp/index.html | 12 ++++-
src/main/webapp/installation-guide.html | 9 ++--
src/main/webapp/mainpage.html | 3 ++
.../webapp/pdf/CarbonData Documentation.pdf | Bin 239761 -> 240561 bytes
src/main/webapp/preaggregate-datamap-guide.html | 7 ++-
src/main/webapp/quick-start-guide.html | 3 ++
src/main/webapp/security.html | 3 ++
src/main/webapp/streaming-guide.html | 3 ++
.../supported-data-types-in-carbondata.html | 3 ++
src/main/webapp/timeseries-datamap-guide.html | 3 ++
src/main/webapp/troubleshooting.html | 3 ++
src/main/webapp/useful-tips-on-carbondata.html | 10 +++++
src/main/webapp/videogallery.html | 4 +-
src/site/markdown/configuration-parameters.md | 20 ++++-----
.../markdown/data-management-on-carbondata.md | 11 +++++
src/site/markdown/faq.md | 28 ++++++------
src/site/markdown/installation-guide.md | 6 +--
src/site/markdown/preaggregate-datamap-guide.md | 4 +-
src/site/markdown/useful-tips-on-carbondata.md | 1 +
src/site/pdf.xml | 2 +-
42 files changed, 264 insertions(+), 125 deletions(-)
----------------------------------------------------------------------
[2/3] carbondata-site git commit: added documentation for release
1.3.1
Posted by ch...@apache.org.
added documentation for release 1.3.1
Project: http://git-wip-us.apache.org/repos/asf/carbondata-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata-site/commit/2e8ae8b5
Tree: http://git-wip-us.apache.org/repos/asf/carbondata-site/tree/2e8ae8b5
Diff: http://git-wip-us.apache.org/repos/asf/carbondata-site/diff/2e8ae8b5
Branch: refs/heads/asf-site
Commit: 2e8ae8b5627ff8cec2046d536e8edbe2ee415ff3
Parents: e4ea53f
Author: vandana <va...@gmail.com>
Authored: Tue Mar 13 16:12:12 2018 +0530
Committer: vandana <va...@gmail.com>
Committed: Tue Mar 13 16:26:33 2018 +0530
----------------------------------------------------------------------
content/configuration-parameters.html | 45 ++++++++-----------
content/data-management-on-carbondata.html | 15 +++++++
content/faq.html | 31 +++++++------
content/file-structure-of-carbondata.html | 3 ++
content/index.html | 12 ++++-
content/installation-guide.html | 9 ++--
content/mainpage.html | 3 ++
content/pdf/CarbonData Documentation.pdf | Bin 239761 -> 240561 bytes
content/preaggregate-datamap-guide.html | 7 ++-
content/quick-start-guide.html | 3 ++
content/security.html | 3 ++
content/streaming-guide.html | 3 ++
content/supported-data-types-in-carbondata.html | 3 ++
content/timeseries-datamap-guide.html | 3 ++
content/troubleshooting.html | 3 ++
content/useful-tips-on-carbondata.html | 10 +++++
content/videogallery.html | 4 +-
src/main/scala/html/header.html | 3 ++
src/main/webapp/configuration-parameters.html | 45 ++++++++-----------
.../webapp/data-management-on-carbondata.html | 15 +++++++
src/main/webapp/faq.html | 31 +++++++------
.../webapp/file-structure-of-carbondata.html | 3 ++
src/main/webapp/index.html | 12 ++++-
src/main/webapp/installation-guide.html | 9 ++--
src/main/webapp/mainpage.html | 3 ++
.../webapp/pdf/CarbonData Documentation.pdf | Bin 239761 -> 240561 bytes
src/main/webapp/preaggregate-datamap-guide.html | 7 ++-
src/main/webapp/quick-start-guide.html | 3 ++
src/main/webapp/security.html | 3 ++
src/main/webapp/streaming-guide.html | 3 ++
.../supported-data-types-in-carbondata.html | 3 ++
src/main/webapp/timeseries-datamap-guide.html | 3 ++
src/main/webapp/troubleshooting.html | 3 ++
src/main/webapp/useful-tips-on-carbondata.html | 10 +++++
src/main/webapp/videogallery.html | 4 +-
src/site/markdown/configuration-parameters.md | 20 ++++-----
.../markdown/data-management-on-carbondata.md | 11 +++++
src/site/markdown/faq.md | 28 ++++++------
src/site/markdown/installation-guide.md | 6 +--
src/site/markdown/preaggregate-datamap-guide.md | 4 +-
src/site/markdown/useful-tips-on-carbondata.md | 1 +
src/site/pdf.xml | 2 +-
42 files changed, 264 insertions(+), 125 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/configuration-parameters.html
----------------------------------------------------------------------
diff --git a/content/configuration-parameters.html b/content/configuration-parameters.html
index 49b580f..4f4f1b0 100644
--- a/content/configuration-parameters.html
+++ b/content/configuration-parameters.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -196,23 +199,23 @@
<tbody>
<tr>
<td>carbon.storelocation</td>
-<td>/user/hive/warehouse/carbon.store</td>
-<td>Location where CarbonData will create the store, and write the data in its own format. NOTE: Store location should be in HDFS.</td>
+<td></td>
+<td>Location where CarbonData will create the store, and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path. NOTE: Store location should be in HDFS.</td>
</tr>
<tr>
<td>carbon.ddl.base.hdfs.url</td>
-<td>hdfs://hacluster/opt/data</td>
+<td></td>
<td>This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv.</td>
</tr>
<tr>
<td>carbon.badRecords.location</td>
-<td>/opt/Carbon/Spark/badrecords</td>
+<td></td>
<td>Path where the bad records are stored.</td>
</tr>
<tr>
<td>carbon.data.file.version</td>
-<td>3</td>
-<td>If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only. The default value for this parameter is 3(latest version is set as default version). It improves the query performance by ~20% to 50%. For configuring V3 format explicitly, add carbon.data.file.version = V3 in carbon.properties file.</td>
+<td>V3</td>
+<td>If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only.</td>
</tr>
<tr>
<td>carbon.streaming.auto.handoff.enabled</td>
@@ -224,6 +227,11 @@
<td>1024000000</td>
<td>This parameter defines the maximum size of the streaming segment. Setting this parameter to appropriate value will avoid impacting the streaming ingestion. The value is in bytes.</td>
</tr>
+<tr>
+<td>carbon.query.show.datamaps</td>
+<td>true</td>
+<td>If this parameter value is set to true, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.</td>
+</tr>
</tbody>
</table>
<h2>
@@ -245,13 +253,13 @@
<tbody>
<tr>
<td>carbon.number.of.cores.while.loading</td>
-<td>6</td>
+<td>2</td>
<td>Number of cores to be used while loading data.</td>
<td></td>
</tr>
<tr>
<td>carbon.sort.size</td>
-<td>500000</td>
+<td>100000</td>
<td>Record count to sort and write intermediate files to temp.</td>
<td></td>
</tr>
@@ -450,8 +458,8 @@
<tbody>
<tr>
<td>carbon.sort.file.write.buffer.size</td>
-<td>16777216</td>
-<td>File write buffer size used during sorting (minValue = 10 KB, maxValue=10MB).</td>
+<td>16384</td>
+<td>File write buffer size used during sorting. Minimum allowed buffer size is 10240 byte and Maximum allowed buffer size is 10485760 byte.</td>
</tr>
<tr>
<td>carbon.lock.type</td>
@@ -474,11 +482,6 @@
<td>csv reading buffer size.</td>
</tr>
<tr>
-<td>high.cardinality.value</td>
-<td>100000</td>
-<td>To identify and apply compression for non-high cardinality columns.</td>
-</tr>
-<tr>
<td>carbon.merge.sort.reader.thread</td>
<td>3</td>
<td>Maximum no of threads used for reading intermediate files for final merging.</td>
@@ -504,16 +507,6 @@
<td>Specifies the interval between the retries to obtain the lock for any operation other than load.</td>
</tr>
<tr>
-<td>carbon.tempstore.location</td>
-<td>/opt/Carbon/TempStoreLoc</td>
-<td>Temporary store location. By default it takes System.getProperty("java.io.tmpdir").</td>
-</tr>
-<tr>
-<td>carbon.load.log.counter</td>
-<td>500000</td>
-<td>Data loading records count logger.</td>
-</tr>
-<tr>
<td>carbon.skip.empty.line</td>
<td>false</td>
<td>Setting this property ignores the empty lines in the CSV file during the data load</td>
@@ -608,7 +601,7 @@
<tbody>
<tr>
<td>carbon.cutOffTimestamp</td>
-<td>1970-01-01 05:30:00</td>
+<td></td>
<td>Sets the start date for calculating the timestamp. Java counts the number of milliseconds from start of "1970-01-01 00:00:00". This property is used to customize the start of position. For example "2000-01-01 00:00:00". The date must be in the form "carbon.timestamp.format".</td>
</tr>
<tr>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/data-management-on-carbondata.html
----------------------------------------------------------------------
diff --git a/content/data-management-on-carbondata.html b/content/data-management-on-carbondata.html
index 05c006b..13ff8fb 100644
--- a/content/data-management-on-carbondata.html
+++ b/content/data-management-on-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -506,6 +509,18 @@ true: CSV file is with file header.</p>
<p>NOTE: Date formats are specified by date pattern strings. The date pattern letters in CarbonData are same as in JAVA. Refer to <a href="http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html" target=_blank rel="nofollow">SimpleDateFormat</a>.</p>
</li>
<li>
+<p><strong>SORT COLUMN BOUNDS:</strong> Range bounds for sort columns.</p>
+<pre><code>OPTIONS('SORT_COLUMN_BOUNDS'='v11,v21,v31;v12,v22,v32;v13,v23,v33')
+</code></pre>
+<p>NOTE:</p>
+<ul>
+<li>SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'.</li>
+<li>Each bound is separated by ';' and each field value in bound is separated by ','.</li>
+<li>Carbondata will use these bounds as ranges to process data concurrently.</li>
+<li>Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'.</li>
+</ul>
+</li>
+<li>
<p><strong>SINGLE_PASS:</strong> Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary.</p>
</li>
</ul>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/faq.html
----------------------------------------------------------------------
diff --git a/content/faq.html b/content/faq.html
index b51b071..e6e0a2b 100644
--- a/content/faq.html
+++ b/content/faq.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -182,7 +185,7 @@
<li><a href="#what-is-carbon-lock-type">What is Carbon Lock Type?</a></li>
<li><a href="#how-to-resolve-abstract-method-error">How to resolve Abstract Method Error?</a></li>
<li><a href="#how-carbon-will-behave-when-execute-insert-operation-in-abnormal-scenarios">How Carbon will behave when execute insert operation in abnormal scenarios?</a></li>
-<li>[Why aggregate query is not fetching data from aggregate table?] (#why-aggregate-query-is-not-fetching-data-from-aggregate-table)</li>
+<li><a href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table">Why aggregate query is not fetching data from aggregate table?</a></li>
</ul>
<h2>
<a id="what-are-bad-records" class="anchor" href="#what-are-bad-records" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>What are Bad Records?</h2>
@@ -277,35 +280,35 @@ id city name
<p>When the column type in carbon table is different from the column specified in select statement. The insert operation will still success, but you may get NULL in result, because NULL will be substitute value when conversion type failed.</p>
<h2>
<a id="why-aggregate-query-is-not-fetching-data-from-aggregate-table" class="anchor" href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Why aggregate query is not fetching data from aggregate table?</h2>
-<p>Following are the aggregate queries that won?t fetch data from aggregate table:</p>
+<p>Following are the aggregate queries that won't fetch data from aggregate table:</p>
<ul>
<li>
<strong>Scenario 1</strong> :
When SubQuery predicate is present in the query.</li>
</ul>
-<p>Example</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
-select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry)
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
+select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry);
</code></pre>
<ul>
<li>
<strong>Scenario 2</strong> :
-When aggregate function along with ?in? filter.</li>
+When aggregate function along with 'in' filter.</li>
</ul>
-<p>Example.</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry, sum(gdp) from gdp21 where cntry in (select ctry from pop1) group by cntry;
</code></pre>
<ul>
<li>
<strong>Scenario 3</strong> :
-When aggregate function having ?join? with Equal filter.</li>
+When aggregate function having 'join' with equal filter.</li>
</ul>
-<p>Example.</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry,sum(gdp) from gdp21,pop1 where cntry=ctry group by cntry;
</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/file-structure-of-carbondata.html
----------------------------------------------------------------------
diff --git a/content/file-structure-of-carbondata.html b/content/file-structure-of-carbondata.html
index 1c8e758..5ea9cbb 100644
--- a/content/file-structure-of-carbondata.html
+++ b/content/file-structure-of-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/index.html
----------------------------------------------------------------------
diff --git a/content/index.html b/content/index.html
index d584d01..3316f63 100644
--- a/content/index.html
+++ b/content/index.html
@@ -54,6 +54,9 @@
class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -322,6 +325,13 @@
</h4>
<div class="linkblock">
<div class="block-row">
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a>
+ <span class="release-date">Mar 2018</span>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.3.1+Release"
+ class="whatsnew" target="_blank">what's new</a>
+ </div>
+ <div class="block-row">
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a>
<span class="release-date">Feb 2018</span>
@@ -478,7 +488,7 @@
to do is:</p>
<ol class="orderlist">
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
target="_blank">Download</a>the latest release.
</li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/installation-guide.html
----------------------------------------------------------------------
diff --git a/content/installation-guide.html b/content/installation-guide.html
index 5f1df57..f6300dc 100644
--- a/content/installation-guide.html
+++ b/content/installation-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -258,7 +261,7 @@
<tr>
<td>carbon.storelocation</td>
<td>NO</td>
-<td>Location where data CarbonData will create the store and write the data in its own format.</td>
+<td>Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
<td>Propose to set HDFS directory</td>
</tr>
@@ -367,7 +370,7 @@ mv carbondata.tar.gz carbonlib/
<tr>
<td>carbon.storelocation</td>
<td>NO</td>
-<td>Location where CarbonData will create the store and write the data in its own format.</td>
+<td>Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
<td>Propose to set HDFS directory</td>
</tr>
@@ -407,7 +410,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
</tr>
<tr>
<td>carbon_store_path</td>
-<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties.</td>
+<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td><code>hdfs://<host_name>:port/user/hive/warehouse/carbon.store</code></td>
</tr>
</tbody>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/mainpage.html
----------------------------------------------------------------------
diff --git a/content/mainpage.html b/content/mainpage.html
index 5f7d924..e597cb4 100644
--- a/content/mainpage.html
+++ b/content/mainpage.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/pdf/CarbonData Documentation.pdf
----------------------------------------------------------------------
diff --git a/content/pdf/CarbonData Documentation.pdf b/content/pdf/CarbonData Documentation.pdf
index 3bdcd47..87879b6 100644
Binary files a/content/pdf/CarbonData Documentation.pdf and b/content/pdf/CarbonData Documentation.pdf differ
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/preaggregate-datamap-guide.html
----------------------------------------------------------------------
diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html
index 819ef3a..ab7beca 100644
--- a/content/preaggregate-datamap-guide.html
+++ b/content/preaggregate-datamap-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -359,7 +362,7 @@ assume that query will be faster on smaller table)</p>
SELECT sex, sum(quantity) from sales GROUP BY sex
-SELECT sum(price), country from sales GROUP BY country
+SELECT avg(price), country from sales GROUP BY country
</code></pre>
<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
@@ -384,7 +387,7 @@ tables are loaded successfully, if one of these loads fails, new data are not vi
as if the load operation is not happened.</p>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
-<p>As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be queried directly.
Queries are to be made on main table. While doing query planning, internally CarbonData will check
associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/quick-start-guide.html
----------------------------------------------------------------------
diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html
index 71ca760..002df87 100644
--- a/content/quick-start-guide.html
+++ b/content/quick-start-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/security.html
----------------------------------------------------------------------
diff --git a/content/security.html b/content/security.html
index 3822acb..826ed4a 100644
--- a/content/security.html
+++ b/content/security.html
@@ -45,6 +45,9 @@
aria-expanded="false">Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/streaming-guide.html
----------------------------------------------------------------------
diff --git a/content/streaming-guide.html b/content/streaming-guide.html
index 43992dd..812f1aa 100644
--- a/content/streaming-guide.html
+++ b/content/streaming-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/supported-data-types-in-carbondata.html
----------------------------------------------------------------------
diff --git a/content/supported-data-types-in-carbondata.html b/content/supported-data-types-in-carbondata.html
index dd4cbc5..43d1071 100644
--- a/content/supported-data-types-in-carbondata.html
+++ b/content/supported-data-types-in-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/timeseries-datamap-guide.html
----------------------------------------------------------------------
diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html
index ce36e72..9b136ca 100644
--- a/content/timeseries-datamap-guide.html
+++ b/content/timeseries-datamap-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/troubleshooting.html
----------------------------------------------------------------------
diff --git a/content/troubleshooting.html b/content/troubleshooting.html
index 107fb23..37a07b4 100644
--- a/content/troubleshooting.html
+++ b/content/troubleshooting.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/useful-tips-on-carbondata.html
----------------------------------------------------------------------
diff --git a/content/useful-tips-on-carbondata.html b/content/useful-tips-on-carbondata.html
index 6df49a7..8b653b4 100644
--- a/content/useful-tips-on-carbondata.html
+++ b/content/useful-tips-on-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -453,6 +456,13 @@ scenarios. After the completion of POC, some of the configurations impacting the
<td>Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading.</td>
<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td>
</tr>
+<tr>
+<td>carbon.load.skewedDataOptimization.enabled</td>
+<td>spark/carbonlib/carbon.properties</td>
+<td>Data loading</td>
+<td>Whether to enable size based block allocation strategy for data loading.</td>
+<td>When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB~1GB.</td>
+</tr>
</tbody>
</table>
<p>Note: If your CarbonData instance is provided only for query, you may specify the property 'spark.speculation=true' which is in conf directory of spark.</p>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/content/videogallery.html
----------------------------------------------------------------------
diff --git a/content/videogallery.html b/content/videogallery.html
index bfa5a49..2dd8c67 100644
--- a/content/videogallery.html
+++ b/content/videogallery.html
@@ -49,9 +49,11 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
-
<li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
target="_blank">Apache CarbonData 1.2.0</a></li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/scala/html/header.html
----------------------------------------------------------------------
diff --git a/src/main/scala/html/header.html b/src/main/scala/html/header.html
index 5ce0b5d..d2ebd58 100644
--- a/src/main/scala/html/header.html
+++ b/src/main/scala/html/header.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/configuration-parameters.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/configuration-parameters.html b/src/main/webapp/configuration-parameters.html
index 49b580f..4f4f1b0 100644
--- a/src/main/webapp/configuration-parameters.html
+++ b/src/main/webapp/configuration-parameters.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -196,23 +199,23 @@
<tbody>
<tr>
<td>carbon.storelocation</td>
-<td>/user/hive/warehouse/carbon.store</td>
-<td>Location where CarbonData will create the store, and write the data in its own format. NOTE: Store location should be in HDFS.</td>
+<td></td>
+<td>Location where CarbonData will create the store, and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path. NOTE: Store location should be in HDFS.</td>
</tr>
<tr>
<td>carbon.ddl.base.hdfs.url</td>
-<td>hdfs://hacluster/opt/data</td>
+<td></td>
<td>This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv.</td>
</tr>
<tr>
<td>carbon.badRecords.location</td>
-<td>/opt/Carbon/Spark/badrecords</td>
+<td></td>
<td>Path where the bad records are stored.</td>
</tr>
<tr>
<td>carbon.data.file.version</td>
-<td>3</td>
-<td>If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only. The default value for this parameter is 3(latest version is set as default version). It improves the query performance by ~20% to 50%. For configuring V3 format explicitly, add carbon.data.file.version = V3 in carbon.properties file.</td>
+<td>V3</td>
+<td>If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only.</td>
</tr>
<tr>
<td>carbon.streaming.auto.handoff.enabled</td>
@@ -224,6 +227,11 @@
<td>1024000000</td>
<td>This parameter defines the maximum size of the streaming segment. Setting this parameter to appropriate value will avoid impacting the streaming ingestion. The value is in bytes.</td>
</tr>
+<tr>
+<td>carbon.query.show.datamaps</td>
+<td>true</td>
+<td>If this parameter value is set to true, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.</td>
+</tr>
</tbody>
</table>
<h2>
@@ -245,13 +253,13 @@
<tbody>
<tr>
<td>carbon.number.of.cores.while.loading</td>
-<td>6</td>
+<td>2</td>
<td>Number of cores to be used while loading data.</td>
<td></td>
</tr>
<tr>
<td>carbon.sort.size</td>
-<td>500000</td>
+<td>100000</td>
<td>Record count to sort and write intermediate files to temp.</td>
<td></td>
</tr>
@@ -450,8 +458,8 @@
<tbody>
<tr>
<td>carbon.sort.file.write.buffer.size</td>
-<td>16777216</td>
-<td>File write buffer size used during sorting (minValue = 10 KB, maxValue=10MB).</td>
+<td>16384</td>
+<td>File write buffer size used during sorting. Minimum allowed buffer size is 10240 byte and Maximum allowed buffer size is 10485760 byte.</td>
</tr>
<tr>
<td>carbon.lock.type</td>
@@ -474,11 +482,6 @@
<td>csv reading buffer size.</td>
</tr>
<tr>
-<td>high.cardinality.value</td>
-<td>100000</td>
-<td>To identify and apply compression for non-high cardinality columns.</td>
-</tr>
-<tr>
<td>carbon.merge.sort.reader.thread</td>
<td>3</td>
<td>Maximum no of threads used for reading intermediate files for final merging.</td>
@@ -504,16 +507,6 @@
<td>Specifies the interval between the retries to obtain the lock for any operation other than load.</td>
</tr>
<tr>
-<td>carbon.tempstore.location</td>
-<td>/opt/Carbon/TempStoreLoc</td>
-<td>Temporary store location. By default it takes System.getProperty("java.io.tmpdir").</td>
-</tr>
-<tr>
-<td>carbon.load.log.counter</td>
-<td>500000</td>
-<td>Data loading records count logger.</td>
-</tr>
-<tr>
<td>carbon.skip.empty.line</td>
<td>false</td>
<td>Setting this property ignores the empty lines in the CSV file during the data load</td>
@@ -608,7 +601,7 @@
<tbody>
<tr>
<td>carbon.cutOffTimestamp</td>
-<td>1970-01-01 05:30:00</td>
+<td></td>
<td>Sets the start date for calculating the timestamp. Java counts the number of milliseconds from start of "1970-01-01 00:00:00". This property is used to customize the start of position. For example "2000-01-01 00:00:00". The date must be in the form "carbon.timestamp.format".</td>
</tr>
<tr>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/data-management-on-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/data-management-on-carbondata.html b/src/main/webapp/data-management-on-carbondata.html
index 05c006b..13ff8fb 100644
--- a/src/main/webapp/data-management-on-carbondata.html
+++ b/src/main/webapp/data-management-on-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -506,6 +509,18 @@ true: CSV file is with file header.</p>
<p>NOTE: Date formats are specified by date pattern strings. The date pattern letters in CarbonData are same as in JAVA. Refer to <a href="http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html" target=_blank rel="nofollow">SimpleDateFormat</a>.</p>
</li>
<li>
+<p><strong>SORT COLUMN BOUNDS:</strong> Range bounds for sort columns.</p>
+<pre><code>OPTIONS('SORT_COLUMN_BOUNDS'='v11,v21,v31;v12,v22,v32;v13,v23,v33')
+</code></pre>
+<p>NOTE:</p>
+<ul>
+<li>SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'.</li>
+<li>Each bound is separated by ';' and each field value in bound is separated by ','.</li>
+<li>Carbondata will use these bounds as ranges to process data concurrently.</li>
+<li>Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'.</li>
+</ul>
+</li>
+<li>
<p><strong>SINGLE_PASS:</strong> Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary.</p>
</li>
</ul>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/faq.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/faq.html b/src/main/webapp/faq.html
index b51b071..e6e0a2b 100644
--- a/src/main/webapp/faq.html
+++ b/src/main/webapp/faq.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -182,7 +185,7 @@
<li><a href="#what-is-carbon-lock-type">What is Carbon Lock Type?</a></li>
<li><a href="#how-to-resolve-abstract-method-error">How to resolve Abstract Method Error?</a></li>
<li><a href="#how-carbon-will-behave-when-execute-insert-operation-in-abnormal-scenarios">How Carbon will behave when execute insert operation in abnormal scenarios?</a></li>
-<li>[Why aggregate query is not fetching data from aggregate table?] (#why-aggregate-query-is-not-fetching-data-from-aggregate-table)</li>
+<li><a href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table">Why aggregate query is not fetching data from aggregate table?</a></li>
</ul>
<h2>
<a id="what-are-bad-records" class="anchor" href="#what-are-bad-records" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>What are Bad Records?</h2>
@@ -277,35 +280,35 @@ id city name
<p>When the column type in carbon table is different from the column specified in select statement. The insert operation will still success, but you may get NULL in result, because NULL will be substitute value when conversion type failed.</p>
<h2>
<a id="why-aggregate-query-is-not-fetching-data-from-aggregate-table" class="anchor" href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Why aggregate query is not fetching data from aggregate table?</h2>
-<p>Following are the aggregate queries that won?t fetch data from aggregate table:</p>
+<p>Following are the aggregate queries that won't fetch data from aggregate table:</p>
<ul>
<li>
<strong>Scenario 1</strong> :
When SubQuery predicate is present in the query.</li>
</ul>
-<p>Example</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
-select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry)
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
+select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry);
</code></pre>
<ul>
<li>
<strong>Scenario 2</strong> :
-When aggregate function along with ?in? filter.</li>
+When aggregate function along with 'in' filter.</li>
</ul>
-<p>Example.</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry, sum(gdp) from gdp21 where cntry in (select ctry from pop1) group by cntry;
</code></pre>
<ul>
<li>
<strong>Scenario 3</strong> :
-When aggregate function having ?join? with Equal filter.</li>
+When aggregate function having 'join' with equal filter.</li>
</ul>
-<p>Example.</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+<p>Example:</p>
+<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry,sum(gdp) from gdp21,pop1 where cntry=ctry group by cntry;
</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/file-structure-of-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/file-structure-of-carbondata.html b/src/main/webapp/file-structure-of-carbondata.html
index 1c8e758..5ea9cbb 100644
--- a/src/main/webapp/file-structure-of-carbondata.html
+++ b/src/main/webapp/file-structure-of-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/index.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/index.html b/src/main/webapp/index.html
index d584d01..3316f63 100644
--- a/src/main/webapp/index.html
+++ b/src/main/webapp/index.html
@@ -54,6 +54,9 @@
class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -322,6 +325,13 @@
</h4>
<div class="linkblock">
<div class="block-row">
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a>
+ <span class="release-date">Mar 2018</span>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.3.1+Release"
+ class="whatsnew" target="_blank">what's new</a>
+ </div>
+ <div class="block-row">
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a>
<span class="release-date">Feb 2018</span>
@@ -478,7 +488,7 @@
to do is:</p>
<ol class="orderlist">
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
target="_blank">Download</a>the latest release.
</li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/installation-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/installation-guide.html b/src/main/webapp/installation-guide.html
index 5f1df57..f6300dc 100644
--- a/src/main/webapp/installation-guide.html
+++ b/src/main/webapp/installation-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -258,7 +261,7 @@
<tr>
<td>carbon.storelocation</td>
<td>NO</td>
-<td>Location where data CarbonData will create the store and write the data in its own format.</td>
+<td>Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
<td>Propose to set HDFS directory</td>
</tr>
@@ -367,7 +370,7 @@ mv carbondata.tar.gz carbonlib/
<tr>
<td>carbon.storelocation</td>
<td>NO</td>
-<td>Location where CarbonData will create the store and write the data in its own format.</td>
+<td>Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
<td>Propose to set HDFS directory</td>
</tr>
@@ -407,7 +410,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
</tr>
<tr>
<td>carbon_store_path</td>
-<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties.</td>
+<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path.</td>
<td><code>hdfs://<host_name>:port/user/hive/warehouse/carbon.store</code></td>
</tr>
</tbody>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/mainpage.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/mainpage.html b/src/main/webapp/mainpage.html
index 5f7d924..e597cb4 100644
--- a/src/main/webapp/mainpage.html
+++ b/src/main/webapp/mainpage.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/pdf/CarbonData Documentation.pdf
----------------------------------------------------------------------
diff --git a/src/main/webapp/pdf/CarbonData Documentation.pdf b/src/main/webapp/pdf/CarbonData Documentation.pdf
index 3bdcd47..87879b6 100644
Binary files a/src/main/webapp/pdf/CarbonData Documentation.pdf and b/src/main/webapp/pdf/CarbonData Documentation.pdf differ
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/preaggregate-datamap-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/preaggregate-datamap-guide.html b/src/main/webapp/preaggregate-datamap-guide.html
index 819ef3a..ab7beca 100644
--- a/src/main/webapp/preaggregate-datamap-guide.html
+++ b/src/main/webapp/preaggregate-datamap-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -359,7 +362,7 @@ assume that query will be faster on smaller table)</p>
SELECT sex, sum(quantity) from sales GROUP BY sex
-SELECT sum(price), country from sales GROUP BY country
+SELECT avg(price), country from sales GROUP BY country
</code></pre>
<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
@@ -384,7 +387,7 @@ tables are loaded successfully, if one of these loads fails, new data are not vi
as if the load operation is not happened.</p>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
-<p>As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be queried directly.
Queries are to be made on main table. While doing query planning, internally CarbonData will check
associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/quick-start-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/quick-start-guide.html b/src/main/webapp/quick-start-guide.html
index 71ca760..002df87 100644
--- a/src/main/webapp/quick-start-guide.html
+++ b/src/main/webapp/quick-start-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/security.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/security.html b/src/main/webapp/security.html
index 3822acb..826ed4a 100644
--- a/src/main/webapp/security.html
+++ b/src/main/webapp/security.html
@@ -45,6 +45,9 @@
aria-expanded="false">Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/streaming-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/streaming-guide.html b/src/main/webapp/streaming-guide.html
index 43992dd..812f1aa 100644
--- a/src/main/webapp/streaming-guide.html
+++ b/src/main/webapp/streaming-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/supported-data-types-in-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/supported-data-types-in-carbondata.html b/src/main/webapp/supported-data-types-in-carbondata.html
index dd4cbc5..43d1071 100644
--- a/src/main/webapp/supported-data-types-in-carbondata.html
+++ b/src/main/webapp/supported-data-types-in-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/timeseries-datamap-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/timeseries-datamap-guide.html b/src/main/webapp/timeseries-datamap-guide.html
index ce36e72..9b136ca 100644
--- a/src/main/webapp/timeseries-datamap-guide.html
+++ b/src/main/webapp/timeseries-datamap-guide.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/troubleshooting.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/troubleshooting.html b/src/main/webapp/troubleshooting.html
index 107fb23..37a07b4 100644
--- a/src/main/webapp/troubleshooting.html
+++ b/src/main/webapp/troubleshooting.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/useful-tips-on-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/useful-tips-on-carbondata.html b/src/main/webapp/useful-tips-on-carbondata.html
index 6df49a7..8b653b4 100644
--- a/src/main/webapp/useful-tips-on-carbondata.html
+++ b/src/main/webapp/useful-tips-on-carbondata.html
@@ -51,6 +51,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
<li>
@@ -453,6 +456,13 @@ scenarios. After the completion of POC, some of the configurations impacting the
<td>Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading.</td>
<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td>
</tr>
+<tr>
+<td>carbon.load.skewedDataOptimization.enabled</td>
+<td>spark/carbonlib/carbon.properties</td>
+<td>Data loading</td>
+<td>Whether to enable size based block allocation strategy for data loading.</td>
+<td>When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB~1GB.</td>
+</tr>
</tbody>
</table>
<p>Note: If your CarbonData instance is provided only for query, you may specify the property 'spark.speculation=true' which is in conf directory of spark.</p>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/main/webapp/videogallery.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/videogallery.html b/src/main/webapp/videogallery.html
index bfa5a49..2dd8c67 100644
--- a/src/main/webapp/videogallery.html
+++ b/src/main/webapp/videogallery.html
@@ -49,9 +49,11 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
target="_blank">Apache CarbonData 1.3.0</a></li>
-
<li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
target="_blank">Apache CarbonData 1.2.0</a></li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration-parameters.md b/src/site/markdown/configuration-parameters.md
index 7221420..c4724cc 100644
--- a/src/site/markdown/configuration-parameters.md
+++ b/src/site/markdown/configuration-parameters.md
@@ -32,12 +32,13 @@ This section provides the details of all the configurations required for the Car
| Property | Default Value | Description |
|----------------------------|-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| carbon.storelocation | /user/hive/warehouse/carbon.store | Location where CarbonData will create the store, and write the data in its own format. NOTE: Store location should be in HDFS. |
-| carbon.ddl.base.hdfs.url | hdfs://hacluster/opt/data | This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv. |
-| carbon.badRecords.location | /opt/Carbon/Spark/badrecords | Path where the bad records are stored. |
-| carbon.data.file.version | 3 | If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only. The default value for this parameter is 3(latest version is set as default version). It improves the query performance by ~20% to 50%. For configuring V3 format explicitly, add carbon.data.file.version = V3 in carbon.properties file. |
+| carbon.storelocation | | Location where CarbonData will create the store, and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path. NOTE: Store location should be in HDFS. |
+| carbon.ddl.base.hdfs.url | | This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv. |
+| carbon.badRecords.location | | Path where the bad records are stored. |
+| carbon.data.file.version | V3 | If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only.|
| carbon.streaming.auto.handoff.enabled | true | If this parameter value is set to true, auto trigger handoff function will be enabled.|
| carbon.streaming.segment.max.size | 1024000000 | This parameter defines the maximum size of the streaming segment. Setting this parameter to appropriate value will avoid impacting the streaming ingestion. The value is in bytes.|
+| carbon.query.show.datamaps | true | If this parameter value is set to true, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list. |
## Performance Configuration
This section provides the details of all the configurations required for CarbonData Performance Optimization.
@@ -48,8 +49,8 @@ This section provides the details of all the configurations required for CarbonD
| Parameter | Default Value | Description | Range |
|--------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| carbon.number.of.cores.while.loading | 6 | Number of cores to be used while loading data. | |
-| carbon.sort.size | 500000 | Record count to sort and write intermediate files to temp. | |
+| carbon.number.of.cores.while.loading | 2 | Number of cores to be used while loading data. | |
+| carbon.sort.size | 100000 | Record count to sort and write intermediate files to temp. | |
| carbon.max.driver.lru.cache.size | -1 | Max LRU cache size upto which data will be loaded at the driver side. This value is expressed in MB. Default value of -1 means there is no memory limit for caching. Only integer values greater than 0 are accepted. | |
| carbon.max.executor.lru.cache.size | -1 | Max LRU cache size upto which data will be loaded at the executor side. This value is expressed in MB. Default value of -1 means there is no memory limit for caching. Only integer values greater than 0 are accepted. If this parameter is not configured, then the carbon.max.driver.lru.cache.size value will be considered. | |
| carbon.merge.sort.prefetch | true | Enable prefetch of data during merge sort while reading data from sort temp files in data loading. | |
@@ -98,19 +99,16 @@ This section provides the details of all the configurations required for CarbonD
| Parameter | Default Value | Description |
|---------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| carbon.sort.file.write.buffer.size | 16777216 | File write buffer size used during sorting (minValue = 10 KB, maxValue=10MB). |
+| carbon.sort.file.write.buffer.size | 16384 | File write buffer size used during sorting. Minimum allowed buffer size is 10240 byte and Maximum allowed buffer size is 10485760 byte. |
| carbon.lock.type | LOCALLOCK | This configuration specifies the type of lock to be acquired during concurrent operations on table. There are following types of lock implementation: - LOCALLOCK: Lock is created on local file system as file. This lock is useful when only one spark driver (thrift server) runs on a machine and no other CarbonData spark application is launched concurrently. - HDFSLOCK: Lock is created on HDFS file system as file. This lock is useful when multiple CarbonData spark applications are launched and no ZooKeeper is running on cluster and HDFS supports file based locking. |
| carbon.sort.intermediate.files.limit | 20 | Minimum number of intermediate files after which merged sort can be started (minValue = 2, maxValue=50). |
| carbon.block.meta.size.reserved.percentage | 10 | Space reserved in percentage for writing block meta data in CarbonData file. |
| carbon.csv.read.buffersize.byte | 1048576 | csv reading buffer size. |
-| high.cardinality.value | 100000 | To identify and apply compression for non-high cardinality columns. |
| carbon.merge.sort.reader.thread | 3 | Maximum no of threads used for reading intermediate files for final merging. |
| carbon.concurrent.lock.retries | 100 | Specifies the maximum number of retries to obtain the lock for concurrent operations. This is used for concurrent loading. |
| carbon.concurrent.lock.retry.timeout.sec | 1 | Specifies the interval between the retries to obtain the lock for concurrent operations. |
| carbon.lock.retries | 3 | Specifies the maximum number of retries to obtain the lock for any operations other than load. |
| carbon.lock.retry.timeout.sec | 5 | Specifies the interval between the retries to obtain the lock for any operation other than load. |
-| carbon.tempstore.location | /opt/Carbon/TempStoreLoc | Temporary store location. By default it takes System.getProperty("java.io.tmpdir"). |
-| carbon.load.log.counter | 500000 | Data loading records count logger. |
| carbon.skip.empty.line | false | Setting this property ignores the empty lines in the CSV file during the data load |
| carbon.enable.calculate.size | true | **For Load Operation**: Setting this property calculates the size of the carbon data file (.carbondata) and carbon index file (.carbonindex) for every load and updates the table status file. **For Describe Formatted**: Setting this property calculates the total size of the carbon data files and carbon index files for the respective table and displays in describe formatted command. |
@@ -139,7 +137,7 @@ This section provides the details of all the configurations required for CarbonD
| Parameter | Default Value | Description |
|---------------------------------------|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| carbon.cutOffTimestamp | 1970-01-01 05:30:00 | Sets the start date for calculating the timestamp. Java counts the number of milliseconds from start of "1970-01-01 00:00:00". This property is used to customize the start of position. For example "2000-01-01 00:00:00". The date must be in the form "carbon.timestamp.format". |
+| carbon.cutOffTimestamp | | Sets the start date for calculating the timestamp. Java counts the number of milliseconds from start of "1970-01-01 00:00:00". This property is used to customize the start of position. For example "2000-01-01 00:00:00". The date must be in the form "carbon.timestamp.format". |
| carbon.timegranularity | SECOND | The property used to set the data granularity level DAY, HOUR, MINUTE, or SECOND. |
## Spark Configuration
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/data-management-on-carbondata.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/data-management-on-carbondata.md b/src/site/markdown/data-management-on-carbondata.md
index 2aa4a49..cb9a17c 100644
--- a/src/site/markdown/data-management-on-carbondata.md
+++ b/src/site/markdown/data-management-on-carbondata.md
@@ -418,6 +418,17 @@ This tutorial is going to introduce all commands and data operations on CarbonDa
```
NOTE: Date formats are specified by date pattern strings. The date pattern letters in CarbonData are same as in JAVA. Refer to [SimpleDateFormat](http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html).
+ - **SORT COLUMN BOUNDS:** Range bounds for sort columns.
+
+ ```
+ OPTIONS('SORT_COLUMN_BOUNDS'='v11,v21,v31;v12,v22,v32;v13,v23,v33')
+ ```
+ NOTE:
+ * SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'.
+ * Each bound is separated by ';' and each field value in bound is separated by ','.
+ * Carbondata will use these bounds as ranges to process data concurrently.
+ * Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'.
+
- **SINGLE_PASS:** Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary.
This option specifies whether to use single pass for loading data or not. By default this option is set to FALSE.
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/faq.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/faq.md b/src/site/markdown/faq.md
index 8f04e4f..b5f8254 100644
--- a/src/site/markdown/faq.md
+++ b/src/site/markdown/faq.md
@@ -25,7 +25,7 @@
* [What is Carbon Lock Type?](#what-is-carbon-lock-type)
* [How to resolve Abstract Method Error?](#how-to-resolve-abstract-method-error)
* [How Carbon will behave when execute insert operation in abnormal scenarios?](#how-carbon-will-behave-when-execute-insert-operation-in-abnormal-scenarios)
-* [Why aggregate query is not fetching data from aggregate table?] (#why-aggregate-query-is-not-fetching-data-from-aggregate-table)
+* [Why aggregate query is not fetching data from aggregate table?](#why-aggregate-query-is-not-fetching-data-from-aggregate-table)
## What are Bad Records?
Records that fail to get loaded into the CarbonData due to data type incompatibility or are empty or have incompatible format are classified as Bad Records.
@@ -143,38 +143,38 @@ INSERT INTO TABLE carbon_table SELECT id, city FROM source_table;
When the column type in carbon table is different from the column specified in select statement. The insert operation will still success, but you may get NULL in result, because NULL will be substitute value when conversion type failed.
## Why aggregate query is not fetching data from aggregate table?
-Following are the aggregate queries that won’t fetch data from aggregate table:
+Following are the aggregate queries that won't fetch data from aggregate table:
- **Scenario 1** :
When SubQuery predicate is present in the query.
-Example
+Example:
```
-create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
-select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry)
+create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
+select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry);
```
- **Scenario 2** :
-When aggregate function along with ‘in’ filter.
+When aggregate function along with 'in' filter.
-Example.
+Example:
```
-create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry, sum(gdp) from gdp21 where cntry in (select ctry from pop1) group by cntry;
```
- **Scenario 3** :
-When aggregate function having ‘join’ with Equal filter.
+When aggregate function having 'join' with equal filter.
-Example.
+Example:
```
-create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata'
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp group by ctry;
+create table gdp21(cntry smallint, gdp double, y_year date) stored by 'carbondata';
+create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
select cntry,sum(gdp) from gdp21,pop1 where cntry=ctry group by cntry;
```
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2e8ae8b5/src/site/markdown/installation-guide.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/installation-guide.md b/src/site/markdown/installation-guide.md
index 0c8790b..37f1fdb 100644
--- a/src/site/markdown/installation-guide.md
+++ b/src/site/markdown/installation-guide.md
@@ -61,7 +61,7 @@ followed by :
| Property | Required | Description | Example | Remark |
|----------------------|----------|----------------------------------------------------------------------------------------|-------------------------------------|---------|
-| carbon.storelocation | NO | Location where data CarbonData will create the store and write the data in its own format. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory |
+| carbon.storelocation | NO | Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory |
8. Verify the installation. For example:
@@ -119,7 +119,7 @@ mv carbondata.tar.gz carbonlib/
| Property | Required | Description | Example | Default Value |
|----------------------|----------|----------------------------------------------------------------------------------------|-------------------------------------|---------------|
-| carbon.storelocation | NO | Location where CarbonData will create the store and write the data in its own format. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory|
+| carbon.storelocation | NO | Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.| hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory|
6. Verify the installation.
@@ -148,7 +148,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
| Parameter | Description | Example |
|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|
| CARBON_ASSEMBLY_JAR | CarbonData assembly jar name present in the `$SPARK_HOME/carbonlib/` folder. | carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar |
-| carbon_store_path | This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. | `hdfs://<host_name>:port/user/hive/warehouse/carbon.store` |
+| carbon_store_path | This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path. | `hdfs://<host_name>:port/user/hive/warehouse/carbon.store` |
**NOTE**: From Spark 1.6, by default the Thrift server runs in multi-session mode. Which means each JDBC/ODBC connection owns a copy of their own SQL configuration and temporary function registry. Cached tables are still shared though. If you prefer to run the Thrift server in single-session mode and share all SQL configuration and temporary function registry, please set option `spark.sql.hive.thriftServer.singleSession` to `true`. You may either add this option to `spark-defaults.conf`, or pass it to `spark-submit.sh` via `--conf`: