You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ch...@apache.org on 2021/07/27 14:46:30 UTC
[carbondata-site] branch asf-site updated: reset english website
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/carbondata-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8fbbf17 reset english website
8fbbf17 is described below
commit 8fbbf17e56b8eeebe558260138cbce0237f463a0
Author: chenliang613 <ch...@huawei.com>
AuthorDate: Tue Jul 27 22:46:16 2021 +0800
reset english website
---
content/WEB-INF/classes/application.conf | 1 +
content/bloomfilter-datamap-guide.html | 392 ++++++++++++++++++++++++
content/datamap-developer-guide.html | 266 ++++++++++++++++
content/datamap-management.html | 394 ++++++++++++++++++++++++
content/lucene-datamap-guide.html | 389 ++++++++++++++++++++++++
content/preaggregate-datamap-guide.html | 502 +++++++++++++++++++++++++++++++
content/timeseries-datamap-guide.html | 369 +++++++++++++++++++++++
7 files changed, 2313 insertions(+)
diff --git a/content/WEB-INF/classes/application.conf b/content/WEB-INF/classes/application.conf
index 970edf4..0cc31c3 100644
--- a/content/WEB-INF/classes/application.conf
+++ b/content/WEB-INF/classes/application.conf
@@ -29,6 +29,7 @@ fileList=["configuration-parameters",
"prestosql-guide",
"scd-and-cdc-guide",
"spatial-index-guide",
+ "clean-files",
]
diff --git a/content/bloomfilter-datamap-guide.html b/content/bloomfilter-datamap-guide.html
new file mode 100644
index 0000000..19ee42a
--- /dev/null
+++ b/content/bloomfilter-datamap-guide.html
@@ -0,0 +1,392 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-bloomfilter-datamap" class="anchor" href="#carbondata-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter DataMap</h1>
+<ul>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#bloomfilter-datamap-introduction">BloomFilter Datamap Introduction</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#data-management-with-bloomfilter-datamap">Data Management</a></li>
+<li><a href="#useful-tips">Useful Tips</a></li>
+</ul>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
+<p>Creating BloomFilter DataMap</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING 'bloomfilter'
+DMPROPERTIES ('index_columns'='city, name', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
+</code></pre>
+<p>Dropping specified datamap</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>Showing all DataMaps on this table</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>Disable Datamap</p>
+<blockquote>
+<p>The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
+</blockquote>
+<pre><code>// disable the datamap
+SET carbon.datamap.visible.dbName.tableName.dataMapName = false
+// enable the datamap
+SET carbon.datamap.visible.dbName.tableName.dataMapName = true
+</code></pre>
+<h2>
+<a id="bloomfilter-datamap-introduction" class="anchor" href="#bloomfilter-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter DataMap Introduction</h2>
+<p>A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.
+Carbondata introduced BloomFilter as an index datamap to enhance the performance of querying with precise value.
+It is well suitable for queries that do precise match on high cardinality columns(such as Name/ID).
+Internally, CarbonData maintains a BloomFilter per blocklet for each index column to indicate that whether a value of the column is in this blocklet.
+Just like the other datamaps, BloomFilter datamap is managed along with main tables by CarbonData.
+User can create BloomFilter datamap on specified columns with specified BloomFilter configurations such as size and probability.</p>
+<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
+<pre><code>CREATE TABLE datamap_test (
+ id string,
+ name string,
+ age int,
+ city string,
+ country string)
+STORED AS carbondata
+TBLPROPERTIES('SORT_COLUMNS'='id')
+</code></pre>
+<p>In the above example, <code>id</code> and <code>name</code> are high cardinality columns
+and we always query on <code>id</code> and <code>name</code> with precise value.
+since <code>id</code> is in the sort_columns and it is orderd,
+query on it will be fast because CarbonData can skip all the irrelative blocklets.
+But queries on <code>name</code> may be bad since the blocklet minmax may not help,
+because in each blocklet the range of the value of <code>name</code> may be the same -- all from A* to z*.
+In this case, user can create a BloomFilter datamap on column <code>name</code>.
+Moreover, user can also create a BloomFilter datamap on the sort_columns.
+This is useful if user has too many segments and the range of the value of sort_columns are almost the same.</p>
+<p>User can create BloomFilter datamap using the Create DataMap DDL:</p>
+<pre><code>CREATE DATAMAP dm
+ON TABLE datamap_test
+USING 'bloomfilter'
+DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001', 'BLOOM_COMPRESS'='true')
+</code></pre>
+<p><strong>Properties for BloomFilter DataMap</strong></p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Is Required</th>
+<th>Default Value</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>INDEX_COLUMNS</td>
+<td>YES</td>
+<td></td>
+<td>Carbondata will generate BloomFilter index on these columns. Queries on these columns are usually like 'COL = VAL'.</td>
+</tr>
+<tr>
+<td>BLOOM_SIZE</td>
+<td>NO</td>
+<td>640000</td>
+<td>This value is internally used by BloomFilter as the number of expected insertions, it will affect the size of BloomFilter index. Since each blocklet has a BloomFilter here, so the default value is the approximate distinct index values in a blocklet assuming that each blocklet contains 20 pages and each page contains 32000 records. The value should be an integer.</td>
+</tr>
+<tr>
+<td>BLOOM_FPP</td>
+<td>NO</td>
+<td>0.00001</td>
+<td>This value is internally used by BloomFilter as the False-Positive Probability, it will affect the size of bloomfilter index as well as the number of hash functions for the BloomFilter. The value should be in the range (0, 1). In one test scenario, a 96GB TPCH customer table with bloom_size=320000 and bloom_fpp=0.00001 will result in 18 false positive samples.</td>
+</tr>
+<tr>
+<td>BLOOM_COMPRESS</td>
+<td>NO</td>
+<td>true</td>
+<td>Whether to compress the BloomFilter index files.</td>
+</tr>
+</tbody>
+</table>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data</h2>
+<p>When loading data to main table, BloomFilter files will be generated for all the
+index_columns given in DMProperties which contains the blockletId and a BloomFilter for each index column.
+These index files will be written inside a folder named with datamap name
+inside each segment folders.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying Data</h2>
+<p>User can verify whether a query can leverage BloomFilter datamap by executing <code>EXPLAIN</code> command,
+which will show the transformed logical plan, and thus user can check whether the BloomFilter datamap can skip blocklets during the scan.
+If the datamap does not prune blocklets well, you can try to increase the value of property <code>BLOOM_SIZE</code> and decrease the value of property <code>BLOOM_FPP</code>.</p>
+<h2>
+<a id="data-management-with-bloomfilter-datamap" class="anchor" href="#data-management-with-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management With BloomFilter DataMap</h2>
+<p>Data management with BloomFilter datamap has no difference with that on Lucene datamap.
+You can refer to the corresponding section in <code>CarbonData Lucene DataMap</code>.</p>
+<h2>
+<a id="useful-tips" class="anchor" href="#useful-tips" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Useful Tips</h2>
+<ul>
+<li>BloomFilter DataMap is suggested to be created on the high cardinality columns.
+Query conditions on these columns are always simple <code>equal</code> or <code>in</code>,
+such as 'col1=XX', 'col1 in (XX, YY)'.</li>
+<li>We can create multiple BloomFilter datamaps on one table,
+but we do recommend you to create one BloomFilter datamap that contains multiple index columns,
+because the data loading and query performance will be better.</li>
+<li>
+<code>BLOOM_FPP</code> is only the expected number from user, the actually FPP may be worse.
+If the BloomFilter datamap does not work well,
+you can try to increase <code>BLOOM_SIZE</code> and decrease <code>BLOOM_FPP</code> at the same time.
+Notice that bigger <code>BLOOM_SIZE</code> will increase the size of index file
+and smaller <code>BLOOM_FPP</code> will increase runtime calculation while performing query.</li>
+<li>'0' skipped blocklets of BloomFilter datamap in explain output indicates that
+BloomFilter datamap does not prune better than Main datamap.
+(For example since the data is not ordered, a specific value may be contained in many blocklets. In this case, bloom may not work better than Main DataMap.)
+If this occurs very often, it means that current BloomFilter is useless. You can disable or drop it.
+Sometimes we cannot see any pruning result about BloomFilter datamap in the explain output,
+this indicates that the previous datamap has pruned all the blocklets and there is no need to continue pruning.</li>
+<li>In some scenarios, the BloomFilter datamap may not enhance the query performance significantly
+but if it can reduce the number of spark task,
+there is still a chance that BloomFilter datamap can enhance the performance for concurrent query.</li>
+<li>Note that BloomFilter datamap will decrease the data loading performance and may cause slightly storage expansion (for datamap index file).</li>
+</ul>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+ $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/datamap-developer-guide.html b/content/datamap-developer-guide.html
new file mode 100644
index 0000000..f442fe2
--- /dev/null
+++ b/content/datamap-developer-guide.html
@@ -0,0 +1,266 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="datamap-developer-guide" class="anchor" href="#datamap-developer-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Developer Guide</h1>
+<h3>
+<a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Introduction</h3>
+<p>DataMap is a data structure that can be used to accelerate certain query of the table. Different DataMap can be implemented by developers.
+Currently, there are two 2 types of DataMap supported:</p>
+<ol>
+<li>IndexDataMap: DataMap that leverages index to accelerate filter query</li>
+<li>MVDataMap: DataMap that leverages Materialized View to accelerate olap style query, like SPJG query (select, predicate, join, groupby)</li>
+</ol>
+<h3>
+<a id="datamap-provider" class="anchor" href="#datamap-provider" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Provider</h3>
+<p>When user issues <code>CREATE DATAMAP dm ON TABLE main USING 'provider'</code>, the corresponding DataMapProvider implementation will be created and initialized.
+Currently, the provider string can be:</p>
+<ol>
+<li>preaggregate: A type of MVDataMap that do pre-aggregate of single table</li>
+<li>timeseries: A type of MVDataMap that do pre-aggregate based on time dimension of the table</li>
+<li>class name IndexDataMapFactory implementation: Developer can implement new type of IndexDataMap by extending IndexDataMapFactory</li>
+</ol>
+<p>When user issues <code>DROP DATAMAP dm ON TABLE main</code>, the corresponding DataMapProvider interface will be called.</p>
+<p>Click for more details about <a href="./datamap-management.html#datamap-management">DataMap Management</a> and supported <a href="./datamap-management.html#overview">DSL</a>.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__docs').addClass('selected');
+
+ // Display docs subnav items
+ if (!$('.b-nav__docs').parent().hasClass('nav__item__with__subs--expanded')) {
+ $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/datamap-management.html b/content/datamap-management.html
new file mode 100644
index 0000000..f5f9678
--- /dev/null
+++ b/content/datamap-management.html
@@ -0,0 +1,394 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-datamap-management" class="anchor" href="#carbondata-datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData DataMap Management</h1>
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#automatic-refresh">Automatic Refresh</a></li>
+<li><a href="#manual-refresh">Manual Refresh</a></li>
+<li><a href="#datamap-catalog">DataMap Catalog</a></li>
+<li>
+<a href="#datamap-related-commands">DataMap Related Commands</a>
+<ul>
+<li><a href="#explain">Explain</a></li>
+<li><a href="#show-datamap">Show DataMap</a></li>
+<li><a href="#compaction-on-datamap">Compaction on DataMap</a></li>
+</ul>
+</li>
+</ul>
+<h2>
+<a id="overview" class="anchor" href="#overview" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Overview</h2>
+<p>DataMap can be created using following DDL</p>
+<pre><code> CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ [ON TABLE main_table]
+ USING "datamap_provider"
+ [WITH DEFERRED REBUILD]
+ DMPROPERTIES ('key'='value', ...)
+ AS
+ SELECT statement
+</code></pre>
+<p>Currently, there are 5 DataMap implementations in CarbonData.</p>
+<table>
+<thead>
+<tr>
+<th>DataMap Provider</th>
+<th>Description</th>
+<th>DMPROPERTIES</th>
+<th>Management</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>preaggregate</td>
+<td>single table pre-aggregate table</td>
+<td>No DMPROPERTY is required</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>timeseries</td>
+<td>time dimension rollup table</td>
+<td>event_time, xx_granularity, please refer to <a href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
+</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>mv</td>
+<td>multi-table pre-aggregate table</td>
+<td>No DMPROPERTY is required</td>
+<td>Manual</td>
+</tr>
+<tr>
+<td>lucene</td>
+<td>lucene indexing for text column</td>
+<td>index_columns to specifying the index columns</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>bloomfilter</td>
+<td>bloom filter for high cardinality column, geospatial column</td>
+<td>index_columns to specifying the index columns</td>
+<td>Automatic</td>
+</tr>
+</tbody>
+</table>
+<h2>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h2>
+<p>There are two kinds of management semantic for DataMap.</p>
+<ol>
+<li>Automatic Refresh: Create datamap without <code>WITH DEFERRED REBUILD</code> in the statement, which is by default.</li>
+<li>Manual Refresh: Create datamap with <code>WITH DEFERRED REBUILD</code> in the statement</li>
+</ol>
+<p><strong>CAUTION:</strong>
+If user create MV datamap without specifying <code>WITH DEFERRED REBUILD</code>, carbondata will give a warning and treat the datamap as deferred rebuild.</p>
+<h3>
+<a id="automatic-refresh" class="anchor" href="#automatic-refresh" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Automatic Refresh</h3>
+<p>When user creates a datamap on the main table without using <code>WITH DEFERRED REBUILD</code> syntax, the datamap will be managed by system automatically.
+For every data load to the main table, system will immediately trigger a load to the datamap automatically. These two data loading (to main table and datamap) is executed in a transactional manner, meaning that it will be either both success or neither success.</p>
+<p>The data loading to datamap is incremental based on Segment concept, avoiding a expensive total rebuild.</p>
+<p>If user perform following command on the main table, system will return failure. (reject the operation)</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
+<li>Partition management command: `ALTER TABLE ADD/DROP PARTITION</li>
+</ol>
+<p>If user do want to perform above operations on the main table, user can first drop the datamap, perform the operation, and re-create the datamap again.</p>
+<p>If user drop the main table, the datamap will be dropped immediately too.</p>
+<p>We do recommend you to use this management for index datamap.</p>
+<h3>
+<a id="manual-refresh" class="anchor" href="#manual-refresh" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Manual Refresh</h3>
+<p>When user creates a datamap specifying manual refresh semantic, the datamap is created with status <em>disabled</em> and query will NOT use this datamap until user can issue REBUILD DATAMAP command to build the datamap. For every REBUILD DATAMAP command, system will trigger a full rebuild of the datamap. After rebuild is done, system will change datamap status to <em>enabled</em>, so that it can be used in query rewrite.</p>
+<p>For every new data loading, data update, delete, the related datamap will be made <em>disabled</em>,
+which means that the following queries will not benefit from the datamap before it becomes <em>enabled</em> again.</p>
+<p>If the main table is dropped by user, the related datamap will be dropped immediately.</p>
+<p><strong>Note</strong>:</p>
+<ul>
+<li>If you are creating a datamap on external table, you need to do manual management of the datamap.</li>
+<li>For index datamap such as BloomFilter datamap, there is no need to do manual refresh.
+By default it is automatic refresh,
+which means its data will get refreshed immediately after the datamap is created or the main table is loaded.
+Manual refresh on this datamap will has no impact.</li>
+</ul>
+<h2>
+<a id="datamap-catalog" class="anchor" href="#datamap-catalog" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Catalog</h2>
+<p>Currently, when user creates a datamap, system will store the datamap metadata in a configurable <em>system</em> folder in HDFS or S3.</p>
+<p>In this <em>system</em> folder, it contains:</p>
+<ul>
+<li>DataMapSchema file. It is a json file containing schema for one datamap. Ses DataMapSchema class. If user creates 100 datamaps (on different tables), there will be 100 files in <em>system</em> folder.</li>
+<li>DataMapStatus file. Only one file, it is in json format, and each entry in the file represents for one datamap. Ses DataMapStatusDetail class</li>
+</ul>
+<p>There is a DataMapCatalog interface to retrieve schema of all datamap, it can be used in optimizer to get the metadata of datamap.</p>
+<h2>
+<a id="datamap-related-commands" class="anchor" href="#datamap-related-commands" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Related Commands</h2>
+<h3>
+<a id="explain" class="anchor" href="#explain" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Explain</h3>
+<p>How can user know whether datamap is used in the query?</p>
+<p>User can set enable.query.statistics = true and use EXPLAIN command to know, it will print out something like</p>
+<pre lang="text"><code>== CarbonData Profiler ==
+Hit mv DataMap: datamap1
+Scan Table: default.datamap1_table
++- filter:
++- pruning by CG DataMap
++- all blocklets: 1
+ skipped blocklets: 0
+</code></pre>
+<h3>
+<a id="show-datamap" class="anchor" href="#show-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Show DataMap</h3>
+<p>There is a SHOW DATAMAPS command, when this is issued, system will read all datamap from <em>system</em> folder and print all information on screen. The current information includes:</p>
+<ul>
+<li>DataMapName</li>
+<li>DataMapProviderName like mv, preaggreagte, timeseries, etc</li>
+<li>Associated Table</li>
+</ul>
+<h3>
+<a id="compaction-on-datamap" class="anchor" href="#compaction-on-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compaction on DataMap</h3>
+<p>This feature applies for preaggregate datamap only</p>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong> compact the pre-aggregate tables created on the main table. User need to run Compaction command separately on each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on main table but not performed on pre-aggregate table, all queries still can benefit from pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables can be triggered to merge the segments and files in the pre-aggregate tables.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+ $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/lucene-datamap-guide.html b/content/lucene-datamap-guide.html
new file mode 100644
index 0000000..f461ca5
--- /dev/null
+++ b/content/lucene-datamap-guide.html
@@ -0,0 +1,389 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-lucene-datamap-alpha-feature" class="anchor" href="#carbondata-lucene-datamap-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Lucene DataMap (Alpha Feature)</h1>
+<ul>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#lucene-datamap-introduction">Lucene Datamap</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#data-management-with-lucene-datamap">Data Management</a></li>
+</ul>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
+<p>Lucene DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING 'lucene'
+DMPROPERTIES ('index_columns'='city, name', ...)
+</code></pre>
+<p>DataMap can be dropped using following DDL:</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="lucene-datamap-introduction" class="anchor" href="#lucene-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Lucene DataMap Introduction</h2>
+<p>Lucene is a high performance, full featured text search engine. Lucene is integrated to carbon as
+an index datamap and managed along with main tables by CarbonData. User can create lucene datamap
+to improve query performance on string columns which has content of more length. So, user can
+search tokenized word or pattern of it using lucene query on text content.</p>
+<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
+<pre><code>CREATE TABLE datamap_test (
+ name string,
+ age int,
+ city string,
+ country string)
+STORED AS carbondata
+</code></pre>
+<p>User can create Lucene datamap using the Create DataMap DDL:</p>
+<pre><code>CREATE DATAMAP dm
+ON TABLE datamap_test
+USING 'lucene'
+DMPROPERTIES ('INDEX_COLUMNS' = 'name, country',)
+</code></pre>
+<p><strong>DMProperties</strong></p>
+<ol>
+<li>INDEX_COLUMNS: The list of string columns on which lucene creates indexes.</li>
+<li>FLUSH_CACHE: size of the cache to maintain in Lucene writer, if specified then it tries to
+aggregate the unique data till the cache limit and flush to Lucene. It is best suitable for low
+cardinality dimensions.</li>
+<li>SPLIT_BLOCKLET: when made as true then store the data in blocklet wise in lucene , it means new
+folder will be created for each blocklet, thus, it eliminates storing blockletid in lucene and
+also it makes lucene small chunks of data.</li>
+</ol>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
+<p>When loading data to main table, lucene index files will be generated for all the
+index_columns(String Columns) given in DMProperties which contains information about the data
+location of index_columns. These index files will be written inside a folder named with datamap name
+inside each segment folders.</p>
+<p>A system level configuration carbon.lucene.compression.mode can be added for best compression of
+lucene index files. The default value is speed, where the index writing speed will be more. If the
+value is compression, the index file size will be compressed.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Lucene indexes cannot be queried directly.
+Queries are to be made on main table. when a query with TEXT_MATCH('name:c10') or
+TEXT_MATCH_WITH_LIMIT('name:n10',10)[the second parameter represents the number of result to be
+returned, if user does not specify this value, all results will be returned without any limit] is
+fired, two jobs are fired. The first job writes the temporary files in folder created at table level
+which contains lucene's seach results and these files will be read in second job to give faster
+results. These temporary files will be cleared once the query finishes.</p>
+<p>User can verify whether a query can leverage Lucene datamap or not by executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check whether TEXT_MATCH()
+filter is applied on query or not.</p>
+<p><strong>Note:</strong></p>
+<ol>
+<li>
+<p>The filter columns in TEXT_MATCH or TEXT_MATCH_WITH_LIMIT must be always in lower case and
+filter condition like 'AND','OR' must be in upper case.</p>
+<p>Ex:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+</code></pre>
+</li>
+<li>
+<p>Query supports only one TEXT_MATCH udf for filter condition and not multiple udfs.</p>
+<p>The following query is supported:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+</code></pre>
+<p>The following query is not supported:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10) AND TEXT_MATCH(name:*n*')
+</code></pre>
+</li>
+</ol>
+<p>Below like queries can be converted to text_match queries as following:</p>
+<pre><code>select * from datamap_test where name='n10'
+
+select * from datamap_test where name like 'n1%'
+
+select * from datamap_test where name like '%10'
+
+select * from datamap_test where name like '%n%'
+
+select * from datamap_test where name like '%10' and name not like '%n%'
+</code></pre>
+<p>Lucene TEXT_MATCH Queries:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:n10')
+
+select * from datamap_test where TEXT_MATCH('name:n1*')
+
+select * from datamap_test where TEXT_MATCH('name:*10')
+
+select * from datamap_test where TEXT_MATCH('name:*n*')
+
+select * from datamap_test where TEXT_MATCH('name:*10 -name:*n*')
+</code></pre>
+<p><strong>Note:</strong> For lucene queries and syntax, refer to <a href="http://www.lucenetutorial.com/lucene-query-syntax.html" target=_blank rel="nofollow">lucene-syntax</a></p>
+<h2>
+<a id="data-management-with-lucene-datamap" class="anchor" href="#data-management-with-lucene-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with lucene datamap</h2>
+<p>Once there is lucene datamap is created on the main table, following command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>.</li>
+</ol>
+<p><strong>Note</strong>: Adding a new column is supported, and for dropping columns and change datatype
+command, CarbonData will check whether it will impact the lucene datamap, if not, the operation
+is allowed, otherwise operation will be rejected by throwing exception.</p>
+<ol start="3">
+<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>.</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the lucene datamap by <code>DROP DATAMAP</code> command.</li>
+<li>Carry out the data management operation on main table.</li>
+<li>Create the lucene datamap again by <code>CREATE DATAMAP</code> command.
+Basically, user can manually trigger the operation by re-building the datamap.</li>
+</ol>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+ $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html
new file mode 100644
index 0000000..e4f1c91
--- /dev/null
+++ b/content/preaggregate-datamap-guide.html
@@ -0,0 +1,502 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-pre-aggregate-datamap" class="anchor" href="#carbondata-pre-aggregate-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Pre-aggregate DataMap</h1>
+<ul>
+<li><a href="#quick-example">Quick Example</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#preaggregate-datamap-introduction">Pre-aggregate Table</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
+</ul>
+<h2>
+<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
+<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
+<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
+<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.2</pre></div>
+<p>Start spark-shell in new terminal, type :paste, then copy and run the following code.</p>
+<div class="highlight highlight-source-scala"><pre> <span class="pl-k">import</span> <span class="pl-en">java</span>.<span class="pl-en">io</span>.<span class="pl-en">File</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.{<span class="pl-en">CarbonEnv</span>, <span class="pl-en">SparkSession</span>}
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">CarbonSession</span>.<span class="pl-en">_</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">streaming</span>.{<span class="pl-en">ProcessingTime</span>, <span class="pl-en">StreamingQuery</span>}
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbondata</span>.<span class="pl-en">core</span>.<span class="pl-en">util</span>.<span class="pl-en">path</span>.<span class="pl-en">CarbonStorePath</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">warehouse</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./warehouse<span class="pl-pds">"</span></span>).getCanonicalPath
+ <span class="pl-k">val</span> <span class="pl-smi">metastore</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./metastore<span class="pl-pds">"</span></span>).getCanonicalPath
+
+ <span class="pl-k">val</span> <span class="pl-smi">spark</span> <span class="pl-k">=</span> <span class="pl-en">SparkSession</span>
+ .builder()
+ .master(<span class="pl-s"><span class="pl-pds">"</span>local<span class="pl-pds">"</span></span>)
+ .appName(<span class="pl-s"><span class="pl-pds">"</span>preAggregateExample<span class="pl-pds">"</span></span>)
+ .config(<span class="pl-s"><span class="pl-pds">"</span>spark.sql.warehouse.dir<span class="pl-pds">"</span></span>, warehouse)
+ .getOrCreateCarbonSession(warehouse, metastore)
+
+ spark.sparkContext.setLogLevel(<span class="pl-s"><span class="pl-pds">"</span>ERROR<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> drop table if exists previously</span>
+ spark.sql(s<span class="pl-s"><span class="pl-pds">"</span>DROP TABLE IF EXISTS sales<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create main table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE TABLE sales (</span>
+<span class="pl-s"> | user_id string,</span>
+<span class="pl-s"> | country string,</span>
+<span class="pl-s"> | quantity int,</span>
+<span class="pl-s"> | price bigint)</span>
+<span class="pl-s"> | STORED AS carbondata</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create pre-aggregate table on the main table</span>
+ <span class="pl-c"><span class="pl-c">//</span> If main table already have data, following command </span>
+ <span class="pl-c"><span class="pl-c">//</span> will trigger one immediate load to the pre-aggregate table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE DATAMAP agg_sales</span>
+<span class="pl-s"> | ON TABLE sales</span>
+<span class="pl-s"> | USING "preaggregate"</span>
+<span class="pl-s"> | AS</span>
+<span class="pl-s"> | SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | FROM sales</span>
+<span class="pl-s"> | GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-k">import</span> <span class="pl-en">spark</span>.<span class="pl-en">implicits</span>.<span class="pl-en">_</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">SaveMode</span>
+ <span class="pl-k">import</span> <span class="pl-en">scala</span>.<span class="pl-en">util</span>.<span class="pl-en">Random</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Load data to the main table, it will also</span>
+ <span class="pl-c"><span class="pl-c">//</span> trigger immediate load to pre-aggregate table.</span>
+ <span class="pl-c"><span class="pl-c">//</span> These two loading operation is carried out in a</span>
+ <span class="pl-c"><span class="pl-c">//</span> transactional manner, meaning that the whole </span>
+ <span class="pl-c"><span class="pl-c">//</span> operation will fail if one of the loading fails</span>
+ <span class="pl-k">val</span> <span class="pl-smi">r</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Random</span>()
+ spark.sparkContext.parallelize(<span class="pl-c1">1</span> to <span class="pl-c1">10</span>)
+ .map(x <span class="pl-k">=></span> (<span class="pl-s"><span class="pl-pds">"</span>ID.<span class="pl-pds">"</span></span> <span class="pl-k">+</span> r.nextInt(<span class="pl-c1">100000</span>), <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span> <span class="pl-k">+</span> x <span class="pl-k">%</span> <span class="pl-c1">8</span>, x <span class="pl-k">%</span> <span class="pl-c1">50</span>, x <span class="pl-k">%</span> <span class="pl- [...]
+ .toDF(<span class="pl-s"><span class="pl-pds">"</span>user_id<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>price<span class="pl-pds">"</span></span>)
+ .write
+ .format(<span class="pl-s"><span class="pl-pds">"</span>carbondata<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>tableName<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>sales<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>compress<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>true<span class="pl-pds">"</span></span>)
+ .mode(<span class="pl-en">SaveMode</span>.<span class="pl-en">Append</span>)
+ .save()
+
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> |SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | from sales GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin).show
+
+ spark.stop</pre></div>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
+<p>DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING "datamap_provider"
+DMPROPERTIES ('key'='value', ...)
+AS
+ SELECT statement
+</code></pre>
+<p>The string followed by USING is called DataMap Provider, in this version CarbonData supports two
+kinds of DataMap:</p>
+<ol>
+<li>preaggregate, for pre-aggregate table. Pre-Aggregate table supports two values for DMPROPERTIES.
+a. 'path' is used to specify the store location of the datamap.('path'='/location/').
+b. 'partitioning' when set to false enables user to disable partitioning of the datamap.
+Default value is true for this property.</li>
+<li>timeseries, for timeseries roll-up table. Please refer to <a href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
+</li>
+</ol>
+<p>DataMap can be dropped using following DDL</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="preaggregate-datamap-introduction" class="anchor" href="#preaggregate-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Preaggregate DataMap Introduction</h2>
+<p>Pre-aggregate tables are created as DataMaps and managed as tables internally by CarbonData.
+User can create as many pre-aggregate datamaps required to improve query performance,
+provided the storage requirements and loading speeds are acceptable.</p>
+<p>Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer extension supports to
+select the most efficient pre-aggregate datamap and rewrite the SQL to query against the selected
+datamap instead of the main table. Since the data size of pre-aggregate datamap is smaller,
+user queries are much faster. In our previous experience, we have seen 5X to 100X times faster
+in production SQLs.</p>
+<p>For instance, main table called <strong>sales</strong> which is defined as</p>
+<pre><code>CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+STORED AS carbondata
+</code></pre>
+<p>User can create pre-aggregate tables using the Create DataMap DDL</p>
+<pre><code>CREATE DATAMAP agg_sales
+ON TABLE sales
+USING "preaggregate"
+AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
+</code></pre>
+<h4>
+<a id="functions-supported-in-pre-aggregate-table" class="anchor" href="#functions-supported-in-pre-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Functions supported in pre-aggregate table</h4>
+<table>
+<thead>
+<tr>
+<th>Function</th>
+<th align="center">Rollup supported</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>SUM</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>AVG</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MAX</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MIN</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>COUNT</td>
+<td align="center">Yes</td>
+</tr>
+</tbody>
+</table>
+<h4>
+<a id="how-pre-aggregate-tables-are-selected" class="anchor" href="#how-pre-aggregate-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How pre-aggregate tables are selected</h4>
+<p>When a user query is submitted, during query planning phase, CarbonData will collect all matched
+pre-aggregate tables as candidates according to Relational Algebra transformation rules. Then, the
+best pre-aggregate table for this query will be selected among the candidates based on cost.
+For simplicity, current cost estimation is based on the data size of the pre-aggregate table. (We
+assume that query will be faster on smaller table)</p>
+<p>For the main table <strong>sales</strong> and pre-aggregate table <strong>agg_sales</strong> created above, following queries</p>
+<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT avg(price), country from sales GROUP BY country
+</code></pre>
+<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
+<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
+<p>However, for following queries</p>
+<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT country, max(price) from sales GROUP BY country
+</code></pre>
+<p>will query against main table <strong>sales</strong> only, because it does not satisfy pre-aggregate table
+selection logic.</p>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
+<p>For existing table with loaded data, data load to pre-aggregate table will be triggered by the
+CREATE DATAMAP statement when user creates the pre-aggregate table. For incremental loads after
+aggregates tables are created, loading data to main table triggers the load to pre-aggregate tables
+once main table loading is complete.</p>
+<p>These loads are transactional
+meaning that data on main table and pre-aggregate tables are only visible to the user after all
+tables are loaded successfully, if one of these loads fails, new data are not visible in all tables
+as if the load operation is not happened.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be queried directly.
+Queries are to be made on main table. While doing query planning, internally CarbonData will check
+associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
+<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check whether pre-aggregate
+table is selected.</p>
+<h2>
+<a id="compacting-pre-aggregate-tables" class="anchor" href="#compacting-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting pre-aggregate tables</h2>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong>
+compact the pre-aggregate tables created on the main table. User need to run Compaction command
+separately on each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on
+main table but not performed on pre-aggregate table, all queries still can benefit from
+pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables
+can be triggered to merge the segments and files in the pre-aggregate tables.</p>
+<h2>
+<a id="data-management-with-pre-aggregate-tables" class="anchor" href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with pre-aggregate tables</h2>
+<p>In current implementation, data consistency needs to be maintained for both main table and pre-aggregate
+tables. Once there is pre-aggregate table created on the main table, following command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
+<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>
+</li>
+<li>Complex Datatypes for preaggregate is not supported.</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the pre-aggregate table by <code>DROP DATAMAP</code> command</li>
+<li>Carry out the data management operation on main table</li>
+<li>Create the pre-aggregate table again by <code>CREATE DATAMAP</code> command
+Basically, user can manually trigger the operation by re-building the datamap.</li>
+</ol>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+ $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html
new file mode 100644
index 0000000..a9137d0
--- /dev/null
+++ b/content/timeseries-datamap-guide.html
@@ -0,0 +1,369 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
+ target="_blank">Apache CarbonData 1.4.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"
+ target="_blank">Apache CarbonData 1.4.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"
+ target="_blank">Apache CarbonData 1.3.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-timeseries-datamap" class="anchor" href="#carbondata-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Timeseries DataMap</h1>
+<ul>
+<li><a href="#timeseries-datamap-introduction-alpha-feature">Timeseries DataMap Introduction</a></li>
+<li><a href="#compacting-timeseries-datamp">Compaction</a></li>
+<li><a href="#data-management-on-timeseries-datamap">Data Management</a></li>
+</ul>
+<h2>
+<a id="timeseries-datamap-introduction-alpha-feature" class="anchor" href="#timeseries-datamap-introduction-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap Introduction (Alpha Feature)</h2>
+<p>Timeseries DataMap is a pre-aggregate table implementation based on 'pre-aggregate' DataMap.
+Difference is that Timeseries DataMap has built-in understanding of time hierarchy and
+levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension
+for query.</p>
+<p><strong>CAUTION:</strong> Current version of CarbonData does not support roll-up.It will be implemented in future versions.</p>
+<p>The data loading, querying, compaction command and its behavior is the same as preaggregate DataMap.
+Please refer to <a href="./preaggregate-datamap-guide.html">Pre-aggregate DataMap</a>
+for more information.</p>
+<p>To use this datamap, user can create multiple timeseries datamap on the main table which has
+a <em>event_time</em> column, one datamap for one time granularity.</p>
+<p>For example, below statement effectively create multiple pre-aggregate tables on main table called
+<strong>timeseries</strong></p>
+<pre><code>CREATE DATAMAP agg_year
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'year_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_month
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'month_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_day
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_sales_hour
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>For querying timeseries data, Carbondata has builtin support for following time related UDF</p>
+<pre><code>timeseries(timeseries column name, 'aggregation level')
+</code></pre>
+<pre><code>SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales GROUP BY timeseries(order_time,
+'hour')
+</code></pre>
+<p>It is <strong>not necessary</strong> to create pre-aggregate tables for each granularity unless required for
+query.</p>
+<p>For Example: For main table <strong>sales</strong> , if following timeseries datamaps were created for day
+level and hour level pre-aggregate</p>
+<pre><code> CREATE DATAMAP agg_day
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granularity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+ CREATE DATAMAP agg_sales_hour
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granularity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>Queries like below will not be rolled-up and hit the main table</p>
+<pre><code>Select timeseries(order_time, 'month'), sum(quantity) from sales group by timeseries(order_time,
+ 'month')
+
+Select timeseries(order_time, 'year'), sum(quantity) from sales group by timeseries(order_time,
+ 'year')
+</code></pre>
+<p>NOTE (<b>RESTRICTION</b>):</p>
+<ul>
+<li>Only value of 1 is supported for hierarchy levels. Other hierarchy levels will be supported in
+the future CarbonData release.</li>
+<li>timeseries datamap for the desired levels needs to be created one after the other</li>
+<li>timeseries datamaps created for each level needs to be dropped separately</li>
+</ul>
+<h2>
+<a id="compacting-timeseries-datamp" class="anchor" href="#compacting-timeseries-datamp" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting timeseries datamp</h2>
+<p>Refer to Compaction section in <a href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<h2>
+<a id="data-management-on-timeseries-datamap" class="anchor" href="#data-management-on-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management on timeseries datamap</h2>
+<p>Refer to Data Management section in <a href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+ $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file