You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ch...@apache.org on 2018/03/07 12:23:07 UTC
[1/3] carbondata-site git commit: added md files for datamap and
resolved PDF issue
Repository: carbondata-site
Updated Branches:
refs/heads/asf-site b0888c1b2 -> e4ea53ff1
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/webapp/index.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/index.html b/src/main/webapp/index.html
index f2cbbbf..d584d01 100644
--- a/src/main/webapp/index.html
+++ b/src/main/webapp/index.html
@@ -369,7 +369,7 @@
class="fa fa-github icon-margin-r" aria-hidden="true"></i>
CarbonData Github
</a>
- <a href="pdf/maven-pdf-plugin.pdf" class="quickstart-btn"
+ <a href="pdf/CarbonData Documentation.pdf" class="quickstart-btn"
target="_blank"><i class="fa fa-file-pdf-o icon-margin-r"
aria-hidden="true"></i></i> Download Documentation
</a>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/webapp/mainpage.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/mainpage.html b/src/main/webapp/mainpage.html
index d782c46..5f7d924 100644
--- a/src/main/webapp/mainpage.html
+++ b/src/main/webapp/mainpage.html
@@ -192,9 +192,10 @@
<li><a href="data-management-on-carbondata.html">Data
Management On CarbonData</a></li>
<li><a href="installation-guide.html">Installation Guide</a></li>
- <li><a href="configuration-parameters.html">Configuring
- CarbonData</a></li>
+ <li><a href="configuration-parameters.html">Configuring CarbonData</a></li>
<li><a href="streaming-guide.html">Streaming Guide</a></li>
+ <li><a href="preaggregate-datamap-guide.html">CarbonData Pre-aggregate DataMap</a></li>
+ <li><a href="timeseries-datamap-guide.html">CarbonData Timeseries DataMap</a></li>
<li><a href="faq.html">FAQs</a></li>
<li><a href="troubleshooting.html">Troubleshooting</a></li>
<li><a href="useful-tips-on-carbondata.html">Useful
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/webapp/pdf/CarbonData Documentation.pdf
----------------------------------------------------------------------
diff --git a/src/main/webapp/pdf/CarbonData Documentation.pdf b/src/main/webapp/pdf/CarbonData Documentation.pdf
index f1d564f..3bdcd47 100644
Binary files a/src/main/webapp/pdf/CarbonData Documentation.pdf and b/src/main/webapp/pdf/CarbonData Documentation.pdf differ
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/webapp/preaggregate-datamap-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/preaggregate-datamap-guide.html b/src/main/webapp/preaggregate-datamap-guide.html
new file mode 100644
index 0000000..819ef3a
--- /dev/null
+++ b/src/main/webapp/preaggregate-datamap-guide.html
@@ -0,0 +1,440 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
+ target="_blank">Apache CarbonData 1.2.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"
+ target="_blank">Apache CarbonData 1.1.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"
+ target="_blank">Apache CarbonData 1.1.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"
+ target="_blank">Apache CarbonData 1.0.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"
+ target="_blank">Apache CarbonData 0.2.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"
+ target="_blank">Apache CarbonData 0.1.1</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"
+ target="_blank">Apache CarbonData 0.1.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="mainpage.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="row">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="carbondata-pre-aggregate-datamap" class="anchor" href="#carbondata-pre-aggregate-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Pre-aggregate DataMap</h1>
+<ul>
+<li><a href="#quick-example">Quick Example</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#preaggregate-datamap-introduction">Pre-aggregate Table</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
+</ul>
+<h2>
+<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
+<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
+<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
+<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.2</pre></div>
+<p>Start spark-shell in new terminal, type :paste, then copy and run the following code.</p>
+<div class="highlight highlight-source-scala"><pre> <span class="pl-k">import</span> <span class="pl-smi">java.io.</span><span class="pl-smi">File</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.</span>{<span class="pl-smi">CarbonEnv</span>, <span class="pl-smi">SparkSession</span>}
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.CarbonSession.</span><span class="pl-smi">_</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.streaming.</span>{<span class="pl-smi">ProcessingTime</span>, <span class="pl-smi">StreamingQuery</span>}
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.util.path.</span><span class="pl-smi">CarbonStorePath</span>
+
+ <span class="pl-k">val</span> <span class="pl-en">warehouse</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./warehouse<span class="pl-pds">"</span></span>).getCanonicalPath
+ <span class="pl-k">val</span> <span class="pl-en">metastore</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./metastore<span class="pl-pds">"</span></span>).getCanonicalPath
+
+ <span class="pl-k">val</span> <span class="pl-en">spark</span> <span class="pl-k">=</span> <span class="pl-en">SparkSession</span>
+ .builder()
+ .master(<span class="pl-s"><span class="pl-pds">"</span>local<span class="pl-pds">"</span></span>)
+ .appName(<span class="pl-s"><span class="pl-pds">"</span>preAggregateExample<span class="pl-pds">"</span></span>)
+ .config(<span class="pl-s"><span class="pl-pds">"</span>spark.sql.warehouse.dir<span class="pl-pds">"</span></span>, warehouse)
+ .getOrCreateCarbonSession(warehouse, metastore)
+
+ spark.sparkContext.setLogLevel(<span class="pl-s"><span class="pl-pds">"</span>ERROR<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> drop table if exists previously</span>
+ spark.sql(s<span class="pl-s"><span class="pl-pds">"</span>DROP TABLE IF EXISTS sales<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create main table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE TABLE sales (</span>
+<span class="pl-s"> | user_id string,</span>
+<span class="pl-s"> | country string,</span>
+<span class="pl-s"> | quantity int,</span>
+<span class="pl-s"> | price bigint)</span>
+<span class="pl-s"> | STORED BY 'carbondata'</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create pre-aggregate table on the main table</span>
+ <span class="pl-c"><span class="pl-c">//</span> If main table already have data, following command </span>
+ <span class="pl-c"><span class="pl-c">//</span> will trigger one immediate load to the pre-aggregate table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE DATAMAP agg_sales</span>
+<span class="pl-s"> | ON TABLE sales</span>
+<span class="pl-s"> | USING "preaggregate"</span>
+<span class="pl-s"> | AS</span>
+<span class="pl-s"> | SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | FROM sales</span>
+<span class="pl-s"> | GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-k">import</span> <span class="pl-smi">spark.implicits.</span><span class="pl-smi">_</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.</span><span class="pl-smi">SaveMode</span>
+ <span class="pl-k">import</span> <span class="pl-smi">scala.util.</span><span class="pl-smi">Random</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Load data to the main table, it will also</span>
+ <span class="pl-c"><span class="pl-c">//</span> trigger immediate load to pre-aggregate table.</span>
+ <span class="pl-c"><span class="pl-c">//</span> These two loading operation is carried out in a</span>
+ <span class="pl-c"><span class="pl-c">//</span> transactional manner, meaning that the whole </span>
+ <span class="pl-c"><span class="pl-c">//</span> operation will fail if one of the loading fails</span>
+ <span class="pl-k">val</span> <span class="pl-en">r</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Random</span>()
+ spark.sparkContext.parallelize(<span class="pl-c1">1</span> to <span class="pl-c1">10</span>)
+ .map(x <span class="pl-k">=></span> (<span class="pl-s"><span class="pl-pds">"</span>ID.<span class="pl-pds">"</span></span> <span class="pl-k">+</span> r.nextInt(<span class="pl-c1">100000</span>), <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span> <span class="pl-k">+</span> x <span class="pl-k">%</span> <span class="pl-c1">8</span>, x <span class="pl-k">%</span> <span class="pl-c1">50</span>, x <span class="pl-k">%</span> <span class="pl-c1">60</span>))
+ .toDF(<span class="pl-s"><span class="pl-pds">"</span>user_id<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>price<span class="pl-pds">"</span></span>)
+ .write
+ .format(<span class="pl-s"><span class="pl-pds">"</span>carbondata<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>tableName<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>sales<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>compress<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>true<span class="pl-pds">"</span></span>)
+ .mode(<span class="pl-en">SaveMode</span>.<span class="pl-en">Append</span>)
+ .save()
+
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> |SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | from sales GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin).show
+
+ spark.stop</pre></div>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
+<p>DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING "datamap_provider"
+DMPROPERTIES ('key'='value', ...)
+AS
+ SELECT statement
+</code></pre>
+<p>The string followed by USING is called DataMap Provider, in this version CarbonData supports two
+kinds of DataMap:</p>
+<ol>
+<li>preaggregate, for pre-aggregate table. No DMPROPERTY is required for this DataMap</li>
+<li>timeseries, for timeseries roll-up table. Please refer to <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.html" target=_blank>Timeseries DataMap</a>
+</li>
+</ol>
+<p>DataMap can be dropped using following DDL</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="preaggregate-datamap-introduction" class="anchor" href="#preaggregate-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Preaggregate DataMap Introduction</h2>
+<p>Pre-aggregate tables are created as DataMaps and managed as tables internally by CarbonData.
+User can create as many pre-aggregate datamaps required to improve query performance,
+provided the storage requirements and loading speeds are acceptable.</p>
+<p>Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer extension supports to
+select the most efficient pre-aggregate datamap and rewrite the SQL to query against the selected
+datamap instead of the main table. Since the data size of pre-aggregate datamap is smaller,
+user queries are much faster. In our previous experience, we have seen 5X to 100X times faster
+in production SQLs.</p>
+<p>For instance, main table called <strong>sales</strong> which is defined as</p>
+<pre><code>CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+STORED BY 'carbondata'
+</code></pre>
+<p>User can create pre-aggregate tables using the Create DataMap DDL</p>
+<pre><code>CREATE DATAMAP agg_sales
+ON TABLE sales
+USING "preaggregate"
+AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
+</code></pre>
+<h4>
+<a id="functions-supported-in-pre-aggregate-table" class="anchor" href="#functions-supported-in-pre-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Functions supported in pre-aggregate table</h4>
+<table>
+<thead>
+<tr>
+<th>Function</th>
+<th align="center">Rollup supported</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>SUM</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>AVG</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MAX</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MIN</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>COUNT</td>
+<td align="center">Yes</td>
+</tr>
+</tbody>
+</table>
+<h4>
+<a id="how-pre-aggregate-tables-are-selected" class="anchor" href="#how-pre-aggregate-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How pre-aggregate tables are selected</h4>
+<p>When a user query is submitted, during query planning phase, CarbonData will collect all matched
+pre-aggregate tables as candidates according to Relational Algebra transformation rules. Then, the
+best pre-aggregate table for this query will be selected among the candidates based on cost.
+For simplicity, current cost estimation is based on the data size of the pre-aggregate table. (We
+assume that query will be faster on smaller table)</p>
+<p>For the main table <strong>sales</strong> and pre-aggregate table <strong>agg_sales</strong> created above, following queries</p>
+<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT sum(price), country from sales GROUP BY country
+</code></pre>
+<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
+<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
+<p>However, for following queries</p>
+<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT country, max(price) from sales GROUP BY country
+</code></pre>
+<p>will query against main table <strong>sales</strong> only, because it does not satisfy pre-aggregate table
+selection logic.</p>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
+<p>For existing table with loaded data, data load to pre-aggregate table will be triggered by the
+CREATE DATAMAP statement when user creates the pre-aggregate table. For incremental loads after
+aggregates tables are created, loading data to main table triggers the load to pre-aggregate tables
+once main table loading is complete.</p>
+<p>These loads are transactional
+meaning that data on main table and pre-aggregate tables are only visible to the user after all
+tables are loaded successfully, if one of these loads fails, new data are not visible in all tables
+as if the load operation is not happened.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+Queries are to be made on main table. While doing query planning, internally CarbonData will check
+associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
+<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check whether pre-aggregate
+table is selected.</p>
+<h2>
+<a id="compacting-pre-aggregate-tables" class="anchor" href="#compacting-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting pre-aggregate tables</h2>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong>
+compact the pre-aggregate tables created on the main table. User need to run Compaction command
+separately on each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on
+main table but not performed on pre-aggregate table, all queries still can benefit from
+pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables
+can be triggered to merge the segments and files in the pre-aggregate tables.</p>
+<h2>
+<a id="data-management-with-pre-aggregate-tables" class="anchor" href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with pre-aggregate tables</h2>
+<p>In current implementation, data consistence need to be maintained for both main table and pre-aggregate
+tables. Once there is pre-aggregate table created on the main table, following command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
+<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>
+</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the pre-aggregate table by <code>DROP DATAMAP</code> command</li>
+<li>Carry out the data management operation on main table</li>
+<li>Create the pre-aggregate table again by <code>CREATE DATAMAP</code> command
+Basically, user can manually trigger the operation by re-building the datamap.</li>
+</ol>
+</div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/webapp/timeseries-datamap-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/timeseries-datamap-guide.html b/src/main/webapp/timeseries-datamap-guide.html
new file mode 100644
index 0000000..ce36e72
--- /dev/null
+++ b/src/main/webapp/timeseries-datamap-guide.html
@@ -0,0 +1,322 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
+ target="_blank">Apache CarbonData 1.2.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"
+ target="_blank">Apache CarbonData 1.1.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"
+ target="_blank">Apache CarbonData 1.1.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"
+ target="_blank">Apache CarbonData 1.0.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"
+ target="_blank">Apache CarbonData 0.2.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"
+ target="_blank">Apache CarbonData 0.1.1</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"
+ target="_blank">Apache CarbonData 0.1.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="mainpage.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="row">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="carbondata-timeseries-datamap" class="anchor" href="#carbondata-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Timeseries DataMap</h1>
+<ul>
+<li><a href="#timeseries-datamap-intoduction-(alpha-feature-in-1.3.0)">Timeseries DataMap</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
+</ul>
+<h2>
+<a id="timeseries-datamap-intoduction-alpha-feature-in-130" class="anchor" href="#timeseries-datamap-intoduction-alpha-feature-in-130" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap Intoduction (Alpha feature in 1.3.0)</h2>
+<p>Timeseries DataMap a pre-aggregate table implementation based on 'preaggregate' DataMap.
+Difference is that Timerseries DataMap has built-in understanding of time hierarchy and
+levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension
+for query.</p>
+<p>The data loading, querying, compaction command and its behavior is the same as preaggregate DataMap.
+Please refer to <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>Pre-aggregate DataMap</a>
+for more information.</p>
+<p>To use this datamap, user can create multiple timeseries datamap on the main table which has
+a <em>event_time</em> column, one datamap for one time granularity. Then Carbondata can do automatic
+roll-up for queries on the main table.</p>
+<p>For example, below statement effectively create multiple pre-aggregate tables on main table called
+<strong>timeseries</strong></p>
+<pre><code>CREATE DATAMAP agg_year
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'year_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_month
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'month_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_day
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_sales_hour
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>For querying timeseries data, Carbondata has builtin support for following time related UDF
+to enable automatically roll-up to the desired aggregation level</p>
+<pre><code>timeseries(timeseries column name, 'aggregation level')
+</code></pre>
+<pre><code>SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales GROUP BY timeseries(order_time,
+'hour')
+</code></pre>
+<p>It is <strong>not necessary</strong> to create pre-aggregate tables for each granularity unless required for
+query. Carbondata can roll-up the data and fetch it.</p>
+<p>For Example: For main table <strong>sales</strong> , if following timeseries datamaps were created for day
+level and hour level pre-aggregate</p>
+<pre><code> CREATE DATAMAP agg_day
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+ CREATE DATAMAP agg_sales_hour
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>Queries like below will be rolled-up and hit the timeseries datamaps</p>
+<pre><code>Select timeseries(order_time, 'month'), sum(quantity) from sales group by timeseries(order_time,
+ 'month')
+
+Select timeseries(order_time, 'year'), sum(quantity) from sales group by timeseries(order_time,
+ 'year')
+</code></pre>
+<p>NOTE (<b>RESTRICTION</b>):</p>
+<ul>
+<li>Only value of 1 is supported for hierarchy levels. Other hierarchy levels will be supported in
+the future CarbonData release.</li>
+<li>timeseries datamap for the desired levels needs to be created one after the other</li>
+<li>timeseries datamaps created for each level needs to be dropped separately</li>
+</ul>
+<h2>
+<a id="compacting-timeseries-datamp" class="anchor" href="#compacting-timeseries-datamp" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting timeseries datamp</h2>
+<p>Refer to Compaction section in <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<h2>
+<a id="data-management-on-timeseries-datamap" class="anchor" href="#data-management-on-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management on timeseries datamap</h2>
+<p>Refer to Data Management section in <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/site/markdown/preaggregate-datamap-guide.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/preaggregate-datamap-guide.md b/src/site/markdown/preaggregate-datamap-guide.md
new file mode 100644
index 0000000..199f674
--- /dev/null
+++ b/src/site/markdown/preaggregate-datamap-guide.md
@@ -0,0 +1,252 @@
+# CarbonData Pre-aggregate DataMap
+
+* [Quick Example](#quick-example)
+* [DataMap Management](#datamap-management)
+* [Pre-aggregate Table](#preaggregate-datamap-introduction)
+* [Loading Data](#loading-data)
+* [Querying Data](#querying-data)
+* [Compaction](#compacting-pre-aggregate-tables)
+* [Data Management](#data-management-with-pre-aggregate-tables)
+
+## Quick example
+Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME
+
+Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars
+```shell
+mvn clean package -DskipTests -Pspark-2.2
+```
+
+Start spark-shell in new terminal, type :paste, then copy and run the following code.
+```scala
+ import java.io.File
+ import org.apache.spark.sql.{CarbonEnv, SparkSession}
+ import org.apache.spark.sql.CarbonSession._
+ import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery}
+ import org.apache.carbondata.core.util.path.CarbonStorePath
+
+ val warehouse = new File("./warehouse").getCanonicalPath
+ val metastore = new File("./metastore").getCanonicalPath
+
+ val spark = SparkSession
+ .builder()
+ .master("local")
+ .appName("preAggregateExample")
+ .config("spark.sql.warehouse.dir", warehouse)
+ .getOrCreateCarbonSession(warehouse, metastore)
+
+ spark.sparkContext.setLogLevel("ERROR")
+
+ // drop table if exists previously
+ spark.sql(s"DROP TABLE IF EXISTS sales")
+
+ // Create main table
+ spark.sql(
+ s"""
+ | CREATE TABLE sales (
+ | user_id string,
+ | country string,
+ | quantity int,
+ | price bigint)
+ | STORED BY 'carbondata'
+ """.stripMargin)
+
+ // Create pre-aggregate table on the main table
+ // If main table already have data, following command
+ // will trigger one immediate load to the pre-aggregate table
+ spark.sql(
+ s"""
+ | CREATE DATAMAP agg_sales
+ | ON TABLE sales
+ | USING "preaggregate"
+ | AS
+ | SELECT country, sum(quantity), avg(price)
+ | FROM sales
+ | GROUP BY country
+ """.stripMargin)
+
+ import spark.implicits._
+ import org.apache.spark.sql.SaveMode
+ import scala.util.Random
+
+ // Load data to the main table, it will also
+ // trigger immediate load to pre-aggregate table.
+ // These two loading operation is carried out in a
+ // transactional manner, meaning that the whole
+ // operation will fail if one of the loading fails
+ val r = new Random()
+ spark.sparkContext.parallelize(1 to 10)
+ .map(x => ("ID." + r.nextInt(100000), "country" + x % 8, x % 50, x % 60))
+ .toDF("user_id", "country", "quantity", "price")
+ .write
+ .format("carbondata")
+ .option("tableName", "sales")
+ .option("compress", "true")
+ .mode(SaveMode.Append)
+ .save()
+
+ spark.sql(
+ s"""
+ |SELECT country, sum(quantity), avg(price)
+ | from sales GROUP BY country
+ """.stripMargin).show
+
+ spark.stop
+```
+
+#### DataMap Management
+DataMap can be created using following DDL
+ ```
+ CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ ON TABLE main_table
+ USING "datamap_provider"
+ DMPROPERTIES ('key'='value', ...)
+ AS
+ SELECT statement
+ ```
+The string followed by USING is called DataMap Provider, in this version CarbonData supports two
+kinds of DataMap:
+1. preaggregate, for pre-aggregate table. No DMPROPERTY is required for this DataMap
+2. timeseries, for timeseries roll-up table. Please refer to [Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.md)
+
+DataMap can be dropped using following DDL
+ ```
+ DROP DATAMAP [IF EXISTS] datamap_name
+ ON TABLE main_table
+ ```
+To show all DataMaps created, use:
+ ```
+ SHOW DATAMAP
+ ON TABLE main_table
+ ```
+It will show all DataMaps created on main table.
+
+
+## Preaggregate DataMap Introduction
+ Pre-aggregate tables are created as DataMaps and managed as tables internally by CarbonData.
+ User can create as many pre-aggregate datamaps required to improve query performance,
+ provided the storage requirements and loading speeds are acceptable.
+
+ Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer extension supports to
+ select the most efficient pre-aggregate datamap and rewrite the SQL to query against the selected
+ datamap instead of the main table. Since the data size of pre-aggregate datamap is smaller,
+ user queries are much faster. In our previous experience, we have seen 5X to 100X times faster
+ in production SQLs.
+
+ For instance, main table called **sales** which is defined as
+
+ ```
+ CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+ STORED BY 'carbondata'
+ ```
+
+ User can create pre-aggregate tables using the Create DataMap DDL
+
+ ```
+ CREATE DATAMAP agg_sales
+ ON TABLE sales
+ USING "preaggregate"
+ AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
+ ```
+
+#### Functions supported in pre-aggregate table
+
+| Function | Rollup supported |
+|----------|:----------------:|
+| SUM |Yes |
+| AVG |Yes |
+| MAX |Yes |
+| MIN |Yes |
+| COUNT |Yes |
+
+
+#### How pre-aggregate tables are selected
+When a user query is submitted, during query planning phase, CarbonData will collect all matched
+pre-aggregate tables as candidates according to Relational Algebra transformation rules. Then, the
+best pre-aggregate table for this query will be selected among the candidates based on cost.
+For simplicity, current cost estimation is based on the data size of the pre-aggregate table. (We
+assume that query will be faster on smaller table)
+
+For the main table **sales** and pre-aggregate table **agg_sales** created above, following queries
+```
+SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT sum(price), country from sales GROUP BY country
+```
+
+will be transformed by CarbonData's query planner to query against pre-aggregate table
+**agg_sales** instead of the main table **sales**
+
+However, for following queries
+```
+SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT country, max(price) from sales GROUP BY country
+```
+
+will query against main table **sales** only, because it does not satisfy pre-aggregate table
+selection logic.
+
+## Loading data
+For existing table with loaded data, data load to pre-aggregate table will be triggered by the
+CREATE DATAMAP statement when user creates the pre-aggregate table. For incremental loads after
+aggregates tables are created, loading data to main table triggers the load to pre-aggregate tables
+once main table loading is complete.
+
+These loads are transactional
+meaning that data on main table and pre-aggregate tables are only visible to the user after all
+tables are loaded successfully, if one of these loads fails, new data are not visible in all tables
+as if the load operation is not happened.
+
+## Querying data
+As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+Queries are to be made on main table. While doing query planning, internally CarbonData will check
+associated pre-aggregate tables with the main table, and do query plan transformation accordingly.
+
+User can verify whether a query can leverage pre-aggregate table or not by executing `EXPLAIN`
+command, which will show the transformed logical plan, and thus user can check whether pre-aggregate
+table is selected.
+
+
+## Compacting pre-aggregate tables
+Running Compaction command (`ALTER TABLE COMPACT`) on main table will **not automatically**
+compact the pre-aggregate tables created on the main table. User need to run Compaction command
+separately on each pre-aggregate table to compact them.
+
+Compaction is an optional operation for pre-aggregate table. If compaction is performed on
+main table but not performed on pre-aggregate table, all queries still can benefit from
+pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables
+can be triggered to merge the segments and files in the pre-aggregate tables.
+
+## Data Management with pre-aggregate tables
+In current implementation, data consistence need to be maintained for both main table and pre-aggregate
+tables. Once there is pre-aggregate table created on the main table, following command on the main
+table
+is not supported:
+1. Data management command: `UPDATE/DELETE/DELETE SEGMENT`.
+2. Schema management command: `ALTER TABLE DROP COLUMN`, `ALTER TABLE CHANGE DATATYPE`,
+`ALTER TABLE RENAME`. Note that adding a new column is supported, and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
+ not, the operation is allowed, otherwise operation will be rejected by throwing exception.
+3. Partition management command: `ALTER TABLE ADD/DROP PARTITION`
+
+However, there is still way to support these operations on main table, in current CarbonData
+release, user can do as following:
+1. Remove the pre-aggregate table by `DROP DATAMAP` command
+2. Carry out the data management operation on main table
+3. Create the pre-aggregate table again by `CREATE DATAMAP` command
+Basically, user can manually trigger the operation by re-building the datamap.
+
+
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/site/markdown/timeseries-datamap-guide.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/timeseries-datamap-guide.md b/src/site/markdown/timeseries-datamap-guide.md
new file mode 100644
index 0000000..886c161
--- /dev/null
+++ b/src/site/markdown/timeseries-datamap-guide.md
@@ -0,0 +1,146 @@
+# CarbonData Timeseries DataMap
+
+* [Timeseries DataMap](#timeseries-datamap-intoduction-(alpha-feature-in-1.3.0))
+* [Compaction](#compacting-pre-aggregate-tables)
+* [Data Management](#data-management-with-pre-aggregate-tables)
+
+## Timeseries DataMap Intoduction (Alpha feature in 1.3.0)
+Timeseries DataMap a pre-aggregate table implementation based on 'preaggregate' DataMap.
+Difference is that Timerseries DataMap has built-in understanding of time hierarchy and
+levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension
+for query.
+
+The data loading, querying, compaction command and its behavior is the same as preaggregate DataMap.
+Please refer to [Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md)
+for more information.
+
+To use this datamap, user can create multiple timeseries datamap on the main table which has
+a *event_time* column, one datamap for one time granularity. Then Carbondata can do automatic
+roll-up for queries on the main table.
+
+For example, below statement effectively create multiple pre-aggregate tables on main table called
+**timeseries**
+
+```
+CREATE DATAMAP agg_year
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'year_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_month
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'month_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_day
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_sales_hour
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+```
+
+For querying timeseries data, Carbondata has builtin support for following time related UDF
+to enable automatically roll-up to the desired aggregation level
+```
+timeseries(timeseries column name, 'aggregation level')
+```
+```
+SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales GROUP BY timeseries(order_time,
+'hour')
+```
+
+It is **not necessary** to create pre-aggregate tables for each granularity unless required for
+query. Carbondata can roll-up the data and fetch it.
+
+For Example: For main table **sales** , if following timeseries datamaps were created for day
+level and hour level pre-aggregate
+
+```
+ CREATE DATAMAP agg_day
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+ CREATE DATAMAP agg_sales_hour
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+```
+
+Queries like below will be rolled-up and hit the timeseries datamaps
+```
+Select timeseries(order_time, 'month'), sum(quantity) from sales group by timeseries(order_time,
+ 'month')
+
+Select timeseries(order_time, 'year'), sum(quantity) from sales group by timeseries(order_time,
+ 'year')
+```
+
+NOTE (<b>RESTRICTION</b>):
+* Only value of 1 is supported for hierarchy levels. Other hierarchy levels will be supported in
+the future CarbonData release.
+* timeseries datamap for the desired levels needs to be created one after the other
+* timeseries datamaps created for each level needs to be dropped separately
+
+
+## Compacting timeseries datamp
+Refer to Compaction section in [preaggregation datamap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md).
+Same applies to timeseries datamap.
+
+## Data Management on timeseries datamap
+Refer to Data Management section in [preaggregation datamap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md).
+Same applies to timeseries datamap.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/site/pdf.xml
----------------------------------------------------------------------
diff --git a/src/site/pdf.xml b/src/site/pdf.xml
index 26831c6..2232a3e 100644
--- a/src/site/pdf.xml
+++ b/src/site/pdf.xml
@@ -16,6 +16,8 @@
<item name="Installation" ref='installation-guide.md'/>
<item name="Configuring CarbonData" ref='configuration-parameters.md'/>
<item name="Streaming" ref='streaming-guide.md'/>
+ <item name="CarbonData Pre-aggregate DataMap" ref='preaggregate-datamap-guide.md'/>
+ <item name="CarbonData Timeseries DataMap" ref='timeseries-datamap-guide.md'/>
<item name="FAQs" ref='faq.md'/>
<item name="Troubleshooting" ref='troubleshooting.md'/>
<item name="Useful Tips" ref='useful-tips-on-carbondata.md'/>
[2/3] carbondata-site git commit: added md files for datamap and
resolved PDF issue
Posted by ch...@apache.org.
added md files for datamap and resolved PDF issue
Project: http://git-wip-us.apache.org/repos/asf/carbondata-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata-site/commit/a4bd7be6
Tree: http://git-wip-us.apache.org/repos/asf/carbondata-site/tree/a4bd7be6
Diff: http://git-wip-us.apache.org/repos/asf/carbondata-site/diff/a4bd7be6
Branch: refs/heads/asf-site
Commit: a4bd7be61c8b0480db97170c611e2df84a63f18c
Parents: b0888c1
Author: vandana <va...@gmail.com>
Authored: Tue Mar 6 22:45:57 2018 +0530
Committer: vandana <va...@gmail.com>
Committed: Wed Mar 7 12:27:39 2018 +0530
----------------------------------------------------------------------
content/WEB-INF/classes/MDFileConverter.class | Bin 8964 -> 10321 bytes
content/WEB-INF/classes/application.conf | 6 +
content/index.html | 2 +-
content/mainpage.html | 5 +-
content/pdf/CarbonData Documentation.pdf | Bin 224247 -> 239761 bytes
content/preaggregate-datamap-guide.html | 440 ++++++++++++++++
content/release-guide.html | 526 +++++++++++++++++++
content/timeseries-datamap-guide.html | 322 ++++++++++++
src/main/resources/application.conf | 6 +
src/main/scala/MDFileConverter.scala | 73 ++-
src/main/webapp/index.html | 2 +-
src/main/webapp/mainpage.html | 5 +-
.../webapp/pdf/CarbonData Documentation.pdf | Bin 224247 -> 239761 bytes
src/main/webapp/preaggregate-datamap-guide.html | 440 ++++++++++++++++
src/main/webapp/timeseries-datamap-guide.html | 322 ++++++++++++
src/site/markdown/preaggregate-datamap-guide.md | 252 +++++++++
src/site/markdown/timeseries-datamap-guide.md | 146 +++++
src/site/pdf.xml | 2 +
18 files changed, 2514 insertions(+), 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/WEB-INF/classes/MDFileConverter.class
----------------------------------------------------------------------
diff --git a/content/WEB-INF/classes/MDFileConverter.class b/content/WEB-INF/classes/MDFileConverter.class
index 7b74cf9..76a6063 100644
Binary files a/content/WEB-INF/classes/MDFileConverter.class and b/content/WEB-INF/classes/MDFileConverter.class differ
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/WEB-INF/classes/application.conf
----------------------------------------------------------------------
diff --git a/content/WEB-INF/classes/application.conf b/content/WEB-INF/classes/application.conf
index e20ebe4..5fe9837 100644
--- a/content/WEB-INF/classes/application.conf
+++ b/content/WEB-INF/classes/application.conf
@@ -9,6 +9,12 @@ fileList=["configuration-parameters",
"troubleshooting",
"useful-tips-on-carbondata"
]
+dataMapFileList=[
+ "preaggregate-datamap-guide",
+ "timeseries-datamap-guide"]
+
+dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/datamap/"
+
fileListToRetain=["quick-start-guide",
"installation-guide"
]
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/index.html
----------------------------------------------------------------------
diff --git a/content/index.html b/content/index.html
index f2cbbbf..d584d01 100644
--- a/content/index.html
+++ b/content/index.html
@@ -369,7 +369,7 @@
class="fa fa-github icon-margin-r" aria-hidden="true"></i>
CarbonData Github
</a>
- <a href="pdf/maven-pdf-plugin.pdf" class="quickstart-btn"
+ <a href="pdf/CarbonData Documentation.pdf" class="quickstart-btn"
target="_blank"><i class="fa fa-file-pdf-o icon-margin-r"
aria-hidden="true"></i></i> Download Documentation
</a>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/mainpage.html
----------------------------------------------------------------------
diff --git a/content/mainpage.html b/content/mainpage.html
index d782c46..5f7d924 100644
--- a/content/mainpage.html
+++ b/content/mainpage.html
@@ -192,9 +192,10 @@
<li><a href="data-management-on-carbondata.html">Data
Management On CarbonData</a></li>
<li><a href="installation-guide.html">Installation Guide</a></li>
- <li><a href="configuration-parameters.html">Configuring
- CarbonData</a></li>
+ <li><a href="configuration-parameters.html">Configuring CarbonData</a></li>
<li><a href="streaming-guide.html">Streaming Guide</a></li>
+ <li><a href="preaggregate-datamap-guide.html">CarbonData Pre-aggregate DataMap</a></li>
+ <li><a href="timeseries-datamap-guide.html">CarbonData Timeseries DataMap</a></li>
<li><a href="faq.html">FAQs</a></li>
<li><a href="troubleshooting.html">Troubleshooting</a></li>
<li><a href="useful-tips-on-carbondata.html">Useful
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/pdf/CarbonData Documentation.pdf
----------------------------------------------------------------------
diff --git a/content/pdf/CarbonData Documentation.pdf b/content/pdf/CarbonData Documentation.pdf
index f1d564f..3bdcd47 100644
Binary files a/content/pdf/CarbonData Documentation.pdf and b/content/pdf/CarbonData Documentation.pdf differ
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/preaggregate-datamap-guide.html
----------------------------------------------------------------------
diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html
new file mode 100644
index 0000000..819ef3a
--- /dev/null
+++ b/content/preaggregate-datamap-guide.html
@@ -0,0 +1,440 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
+ target="_blank">Apache CarbonData 1.2.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"
+ target="_blank">Apache CarbonData 1.1.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"
+ target="_blank">Apache CarbonData 1.1.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"
+ target="_blank">Apache CarbonData 1.0.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"
+ target="_blank">Apache CarbonData 0.2.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"
+ target="_blank">Apache CarbonData 0.1.1</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"
+ target="_blank">Apache CarbonData 0.1.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="mainpage.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="row">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="carbondata-pre-aggregate-datamap" class="anchor" href="#carbondata-pre-aggregate-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Pre-aggregate DataMap</h1>
+<ul>
+<li><a href="#quick-example">Quick Example</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#preaggregate-datamap-introduction">Pre-aggregate Table</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
+</ul>
+<h2>
+<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
+<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
+<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
+<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.2</pre></div>
+<p>Start spark-shell in new terminal, type :paste, then copy and run the following code.</p>
+<div class="highlight highlight-source-scala"><pre> <span class="pl-k">import</span> <span class="pl-smi">java.io.</span><span class="pl-smi">File</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.</span>{<span class="pl-smi">CarbonEnv</span>, <span class="pl-smi">SparkSession</span>}
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.CarbonSession.</span><span class="pl-smi">_</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.streaming.</span>{<span class="pl-smi">ProcessingTime</span>, <span class="pl-smi">StreamingQuery</span>}
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.util.path.</span><span class="pl-smi">CarbonStorePath</span>
+
+ <span class="pl-k">val</span> <span class="pl-en">warehouse</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./warehouse<span class="pl-pds">"</span></span>).getCanonicalPath
+ <span class="pl-k">val</span> <span class="pl-en">metastore</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./metastore<span class="pl-pds">"</span></span>).getCanonicalPath
+
+ <span class="pl-k">val</span> <span class="pl-en">spark</span> <span class="pl-k">=</span> <span class="pl-en">SparkSession</span>
+ .builder()
+ .master(<span class="pl-s"><span class="pl-pds">"</span>local<span class="pl-pds">"</span></span>)
+ .appName(<span class="pl-s"><span class="pl-pds">"</span>preAggregateExample<span class="pl-pds">"</span></span>)
+ .config(<span class="pl-s"><span class="pl-pds">"</span>spark.sql.warehouse.dir<span class="pl-pds">"</span></span>, warehouse)
+ .getOrCreateCarbonSession(warehouse, metastore)
+
+ spark.sparkContext.setLogLevel(<span class="pl-s"><span class="pl-pds">"</span>ERROR<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> drop table if exists previously</span>
+ spark.sql(s<span class="pl-s"><span class="pl-pds">"</span>DROP TABLE IF EXISTS sales<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create main table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE TABLE sales (</span>
+<span class="pl-s"> | user_id string,</span>
+<span class="pl-s"> | country string,</span>
+<span class="pl-s"> | quantity int,</span>
+<span class="pl-s"> | price bigint)</span>
+<span class="pl-s"> | STORED BY 'carbondata'</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create pre-aggregate table on the main table</span>
+ <span class="pl-c"><span class="pl-c">//</span> If main table already have data, following command </span>
+ <span class="pl-c"><span class="pl-c">//</span> will trigger one immediate load to the pre-aggregate table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE DATAMAP agg_sales</span>
+<span class="pl-s"> | ON TABLE sales</span>
+<span class="pl-s"> | USING "preaggregate"</span>
+<span class="pl-s"> | AS</span>
+<span class="pl-s"> | SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | FROM sales</span>
+<span class="pl-s"> | GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-k">import</span> <span class="pl-smi">spark.implicits.</span><span class="pl-smi">_</span>
+ <span class="pl-k">import</span> <span class="pl-smi">org.apache.spark.sql.</span><span class="pl-smi">SaveMode</span>
+ <span class="pl-k">import</span> <span class="pl-smi">scala.util.</span><span class="pl-smi">Random</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Load data to the main table, it will also</span>
+ <span class="pl-c"><span class="pl-c">//</span> trigger immediate load to pre-aggregate table.</span>
+ <span class="pl-c"><span class="pl-c">//</span> These two loading operation is carried out in a</span>
+ <span class="pl-c"><span class="pl-c">//</span> transactional manner, meaning that the whole </span>
+ <span class="pl-c"><span class="pl-c">//</span> operation will fail if one of the loading fails</span>
+ <span class="pl-k">val</span> <span class="pl-en">r</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Random</span>()
+ spark.sparkContext.parallelize(<span class="pl-c1">1</span> to <span class="pl-c1">10</span>)
+ .map(x <span class="pl-k">=></span> (<span class="pl-s"><span class="pl-pds">"</span>ID.<span class="pl-pds">"</span></span> <span class="pl-k">+</span> r.nextInt(<span class="pl-c1">100000</span>), <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span> <span class="pl-k">+</span> x <span class="pl-k">%</span> <span class="pl-c1">8</span>, x <span class="pl-k">%</span> <span class="pl-c1">50</span>, x <span class="pl-k">%</span> <span class="pl-c1">60</span>))
+ .toDF(<span class="pl-s"><span class="pl-pds">"</span>user_id<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>price<span class="pl-pds">"</span></span>)
+ .write
+ .format(<span class="pl-s"><span class="pl-pds">"</span>carbondata<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>tableName<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>sales<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>compress<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>true<span class="pl-pds">"</span></span>)
+ .mode(<span class="pl-en">SaveMode</span>.<span class="pl-en">Append</span>)
+ .save()
+
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> |SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | from sales GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin).show
+
+ spark.stop</pre></div>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
+<p>DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING "datamap_provider"
+DMPROPERTIES ('key'='value', ...)
+AS
+ SELECT statement
+</code></pre>
+<p>The string followed by USING is called DataMap Provider, in this version CarbonData supports two
+kinds of DataMap:</p>
+<ol>
+<li>preaggregate, for pre-aggregate table. No DMPROPERTY is required for this DataMap</li>
+<li>timeseries, for timeseries roll-up table. Please refer to <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.html" target=_blank>Timeseries DataMap</a>
+</li>
+</ol>
+<p>DataMap can be dropped using following DDL</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="preaggregate-datamap-introduction" class="anchor" href="#preaggregate-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Preaggregate DataMap Introduction</h2>
+<p>Pre-aggregate tables are created as DataMaps and managed as tables internally by CarbonData.
+User can create as many pre-aggregate datamaps required to improve query performance,
+provided the storage requirements and loading speeds are acceptable.</p>
+<p>Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer extension supports to
+select the most efficient pre-aggregate datamap and rewrite the SQL to query against the selected
+datamap instead of the main table. Since the data size of pre-aggregate datamap is smaller,
+user queries are much faster. In our previous experience, we have seen 5X to 100X times faster
+in production SQLs.</p>
+<p>For instance, main table called <strong>sales</strong> which is defined as</p>
+<pre><code>CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+STORED BY 'carbondata'
+</code></pre>
+<p>User can create pre-aggregate tables using the Create DataMap DDL</p>
+<pre><code>CREATE DATAMAP agg_sales
+ON TABLE sales
+USING "preaggregate"
+AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
+</code></pre>
+<h4>
+<a id="functions-supported-in-pre-aggregate-table" class="anchor" href="#functions-supported-in-pre-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Functions supported in pre-aggregate table</h4>
+<table>
+<thead>
+<tr>
+<th>Function</th>
+<th align="center">Rollup supported</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>SUM</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>AVG</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MAX</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MIN</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>COUNT</td>
+<td align="center">Yes</td>
+</tr>
+</tbody>
+</table>
+<h4>
+<a id="how-pre-aggregate-tables-are-selected" class="anchor" href="#how-pre-aggregate-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How pre-aggregate tables are selected</h4>
+<p>When a user query is submitted, during query planning phase, CarbonData will collect all matched
+pre-aggregate tables as candidates according to Relational Algebra transformation rules. Then, the
+best pre-aggregate table for this query will be selected among the candidates based on cost.
+For simplicity, current cost estimation is based on the data size of the pre-aggregate table. (We
+assume that query will be faster on smaller table)</p>
+<p>For the main table <strong>sales</strong> and pre-aggregate table <strong>agg_sales</strong> created above, following queries</p>
+<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT sum(price), country from sales GROUP BY country
+</code></pre>
+<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
+<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
+<p>However, for following queries</p>
+<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT country, max(price) from sales GROUP BY country
+</code></pre>
+<p>will query against main table <strong>sales</strong> only, because it does not satisfy pre-aggregate table
+selection logic.</p>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
+<p>For existing table with loaded data, data load to pre-aggregate table will be triggered by the
+CREATE DATAMAP statement when user creates the pre-aggregate table. For incremental loads after
+aggregates tables are created, loading data to main table triggers the load to pre-aggregate tables
+once main table loading is complete.</p>
+<p>These loads are transactional
+meaning that data on main table and pre-aggregate tables are only visible to the user after all
+tables are loaded successfully, if one of these loads fails, new data are not visible in all tables
+as if the load operation is not happened.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be queries directly.
+Queries are to be made on main table. While doing query planning, internally CarbonData will check
+associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
+<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check whether pre-aggregate
+table is selected.</p>
+<h2>
+<a id="compacting-pre-aggregate-tables" class="anchor" href="#compacting-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting pre-aggregate tables</h2>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong>
+compact the pre-aggregate tables created on the main table. User need to run Compaction command
+separately on each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on
+main table but not performed on pre-aggregate table, all queries still can benefit from
+pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables
+can be triggered to merge the segments and files in the pre-aggregate tables.</p>
+<h2>
+<a id="data-management-with-pre-aggregate-tables" class="anchor" href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with pre-aggregate tables</h2>
+<p>In current implementation, data consistence need to be maintained for both main table and pre-aggregate
+tables. Once there is pre-aggregate table created on the main table, following command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
+<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>
+</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the pre-aggregate table by <code>DROP DATAMAP</code> command</li>
+<li>Carry out the data management operation on main table</li>
+<li>Create the pre-aggregate table again by <code>CREATE DATAMAP</code> command
+Basically, user can manually trigger the operation by re-building the datamap.</li>
+</ol>
+</div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/release-guide.html
----------------------------------------------------------------------
diff --git a/content/release-guide.html b/content/release-guide.html
new file mode 100644
index 0000000..0a66cce
--- /dev/null
+++ b/content/release-guide.html
@@ -0,0 +1,526 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
+ target="_blank">Apache CarbonData 1.2.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"
+ target="_blank">Apache CarbonData 1.1.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"
+ target="_blank">Apache CarbonData 1.1.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"
+ target="_blank">Apache CarbonData 1.0.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"
+ target="_blank">Apache CarbonData 0.2.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"
+ target="_blank">Apache CarbonData 0.1.1</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"
+ target="_blank">Apache CarbonData 0.1.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="mainpage.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="row">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="apache-carbondata-release-guide" class="anchor" href="#apache-carbondata-release-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Apache CarbonData Release Guide</h1>
+<p>Apache CarbonData periodically declares and publishes releases.</p>
+<p>Each release is executed by a <em>Release Manager</em>, who is selected among the CarbonData committers.
+This document describes the process that the Release Manager follows to perform a release. Any
+changes to this process should be discussed and adopted on the
+<a href="mailto:dev@carbondata.apache.org">dev@ mailing list</a>.</p>
+<p>Please remember that publishing software has legal consequences. This guide complements the
+foundation-wide <a href="http://www.apache.org/dev/release.html" target=_blank rel="nofollow">Product Release Policy</a> and <a href="http://www.apache.org/dev/release-distribution" target=_blank rel="nofollow">Release
+Distribution Policy</a>.</p>
+<h2>
+<a id="decide-to-release" class="anchor" href="#decide-to-release" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Decide to release</h2>
+<p>Deciding to release and selecting a Release Manager is the first step of the release process.
+This is a consensus-based decision of the entire community.</p>
+<p>Anybody can propose a release on the dev@ mailing list, giving a solid argument and nominating a
+committer as the Release Manager (including themselves). There's no formal process, no vote
+requirements, and no timing requirements. Any objections should be resolved by consensus before
+starting the release.</p>
+<p><em>Checklist to proceed to next step:</em></p>
+<ol>
+<li>Community agrees to release</li>
+<li>Community selects a Release Manager</li>
+</ol>
+<h2>
+<a id="prepare-for-the-release" class="anchor" href="#prepare-for-the-release" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prepare for the release</h2>
+<p>Before your first release, you should perform one-time configuration steps. This will set up your
+security keys for signing the artifacts and access release repository.</p>
+<p>To prepare for each release, you should audit the project status in the Jira, and do necessary
+bookkeeping. Finally, you should tag a release.</p>
+<h3>
+<a id="one-time-setup-instructions" class="anchor" href="#one-time-setup-instructions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>One-time setup instructions</h3>
+<h4>
+<a id="gpg-key" class="anchor" href="#gpg-key" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>GPG Key</h4>
+<p>You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide
+<a href="https://www.apache.org/dev/release-signing.html" target=_blank rel="nofollow">release signing guidelines</a>. If you don't have
+a GPG key associated with your Apache account, please create one according to the guidelines.</p>
+<p>Determine your Apache GPG key and key ID, as follows:</p>
+<pre><code>gpg --list-keys
+</code></pre>
+<p>This will list your GPG keys. One of these should reflect your Apache account, for example:</p>
+<pre><code>pub 2048R/845E6689 2016-02-23
+uid Nomen Nescio <anonymous@apache.org>
+sub 2048R/BA4D50BE 2016-02-23
+</code></pre>
+<p>Here, the key ID is the 8-digit hex string in the <code>pub</code> line: <code>845E6689</code>.</p>
+<p>Now, add your Apache GPG key to the CarbonData's <code>KEYS</code> file in <code>dev</code> and <code>release</code> repositories
+at <code>dist.apache.org</code>. Follow the instructions listed at the top of these files.</p>
+<p>Configure <code>git</code> to use this key when signing code by giving it your key ID, as follows:</p>
+<pre><code>git config --global user.signingkey 845E6689
+</code></pre>
+<p>You may drop the <code>--global</code> option if you'd prefer to use this key for the current repository only.</p>
+<p>You may wish to start <code>gpg-agent</code> to unlock your GPG key only once using your passphrase.
+Otherwise, you may need to enter this passphrase several times. The setup of <code>gpg-agent</code> varies
+based on operating system, but may be something like this:</p>
+<pre><code>eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)
+export GPG_TTY=$(tty)
+export GPG_AGENT_INFO
+</code></pre>
+<h4>
+<a id="access-to-apache-nexus" class="anchor" href="#access-to-apache-nexus" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Access to Apache Nexus</h4>
+<p>Configure access to the <a href="https://repository.apache.org" target=_blank rel="nofollow">Apache Nexus repository</a>, used for
+staging repository and promote the artifacts to Maven Central.</p>
+<ol>
+<li>You log in with your Apache account.</li>
+<li>Confirm you have appropriate access by finding <code>org.apache.carbondata</code> under <code>Staging Profiles</code>.</li>
+<li>Navigate to your <code>Profile</code> (top right dropdown menu of the page).</li>
+<li>Choose <code>User Token</code> from the dropdown, then click <code>Access User Token</code>. Copy a snippet of the
+Maven XML configuration block.</li>
+<li>Insert this snippet twice into your global Maven <code>settings.xml</code> file, typically <code>${HOME]/ .m2/settings.xml</code>. The end result should look like this, where <code>TOKEN_NAME</code> and <code>TOKEN_PASSWORD</code>
+are your secret tokens:</li>
+</ol>
+<pre><code> <settings>
+ <servers>
+ <server>
+ <id>apache.releases.https</id>
+ <username>TOKEN_NAME</username>
+ <password>TOKEN_PASSWORD</password>
+ </server>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>TOKEN_NAME</username>
+ <password>TOKEN_PASSWORD</password>
+ </server>
+ </servers>
+ </settings>
+</code></pre>
+<h4>
+<a id="create-a-new-version-in-jira" class="anchor" href="#create-a-new-version-in-jira" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a new version in Jira</h4>
+<p>When contributors resolve an issue in Jira, they are tagging it with a release that will contain
+their changes. With the release currently underway, new issues should be resolved against a
+subsequent future release. Therefore, you should create a release item for this subsequent
+release, as follows:</p>
+<ol>
+<li>In Jira, navigate to <code>CarbonData > Administration > Versions</code>.</li>
+<li>Add a new release: choose the next minor version number compared to the one currently
+underway, select today's date as the <code>Start Date</code>, and choose <code>Add</code>.</li>
+</ol>
+<h4>
+<a id="triage-release-blocking-issues-in-jira" class="anchor" href="#triage-release-blocking-issues-in-jira" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Triage release-blocking issues in Jira</h4>
+<p>There could be outstanding release-blocking issues, which should be triaged before proceeding to
+build the release. We track them by assigning a specific <code>Fix Version</code> field even before the
+issue is resolved.</p>
+<p>The list of release-blocking issues is available at the <a href="https://issues.apache.org/jira/browse/CARBONDATA/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel" rel="nofollow">version status page</a>.
+Triage each unresolved issue with one of the following resolutions:</p>
+<ul>
+<li>If the issue has been resolved and Jira was not updated, resolve it accordingly.</li>
+<li>If the issue has not been resolved and it is acceptable to defer until the next release, update
+the <code>Fix Version</code> field to the new version you just created. Please consider discussing this
+with stakeholders and the dev@ mailing list, as appropriate.</li>
+<li>If the issue has not been resolved and it is not acceptable to release until it is fixed, the
+release cannot proceed. Instead, work with the CarbonData community to resolve the issue.</li>
+</ul>
+<h4>
+<a id="review-release-notes-in-jira" class="anchor" href="#review-release-notes-in-jira" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Review Release Notes in Jira</h4>
+<p>Jira automatically generates Release Notes based on the <code>Fix Version</code> applied to the issues.
+Release Notes are intended for CarbonData users (not CarbonData committers/contributors). You
+should ensure that Release Notes are informative and useful.</p>
+<p>Open the release notes from the <a href="https://issues.apache.org/jira/browse/CARBONDATA/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel" rel="nofollow">version status page</a>
+by choosing the release underway and clicking Release Notes.</p>
+<p>You should verify that the issues listed automatically by Jira are appropriate to appear in the
+Release Notes. Specifically, issues should:</p>
+<ul>
+<li>Be appropriate classified as <code>Bug</code>, <code>New Feature</code>, <code>Improvement</code>, etc.</li>
+<li>Represent noteworthy user-facing changes, such as new functionality, backward-incompatible
+changes, or performance improvements.</li>
+<li>Have occurred since the previous release; an issue that was introduced and fixed between
+releases should not appear in the Release Notes.</li>
+<li>Have an issue title that makes sense when read on its own.</li>
+</ul>
+<p>Adjust any of the above properties to the improve clarity and presentation of the Release Notes.</p>
+<h4>
+<a id="verify-that-a-release-build-works" class="anchor" href="#verify-that-a-release-build-works" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Verify that a Release Build works</h4>
+<p>Run <code>mvn clean install -Prelease</code> to ensure that the build processes that are specific to that
+profile are in good shape.</p>
+<p><em>Checklist to proceed to the next step:</em></p>
+<ol>
+<li>Release Manager's GPG key is published to <code>dist.apache.org</code>.</li>
+<li>Release Manager's GPG key is configured in <code>git</code> configuration.</li>
+<li>Release Manager has <code>org.apache.carbondata</code> listed under <code>Staging Profiles</code> in Nexus.</li>
+<li>Release Manager's Nexus User Token is configured in <code>settings.xml</code>.</li>
+<li>Jira release item for the subsequent release has been created.</li>
+<li>There are no release blocking Jira issues.</li>
+<li>Release Notes in Jira have been audited and adjusted.</li>
+</ol>
+<h3>
+<a id="build-a-release" class="anchor" href="#build-a-release" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Build a release</h3>
+<p>Use Maven release plugin to tag and build release artifacts, as follows:</p>
+<pre><code>mvn release:prepare
+</code></pre>
+<p>Use Maven release plugin to stage these artifacts on the Apache Nexus repository, as follows:</p>
+<pre><code>mvn release:perform
+</code></pre>
+<p>Review all staged artifacts. They should contain all relevant parts for each module, including
+<code>pom.xml</code>, jar, test jar, source, etc. Artifact names should follow
+<a href="https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.carbondata%22" rel="nofollow">the existing format</a>
+in which artifact name mirrors directory structure. Carefully review any new artifacts.</p>
+<p>Close the staging repository on Nexus. When prompted for a description, enter "Apache CarbonData
+x.x.x release".</p>
+<h3>
+<a id="stage-source-release-on-distapacheorg" class="anchor" href="#stage-source-release-on-distapacheorg" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Stage source release on dist.apache.org</h3>
+<p>Copy the source release to dev repository on <code>dist.apache.org</code>.</p>
+<ol>
+<li>If you have not already, check out the section of the <code>dev</code> repository on <code>dist.apache.org</code> via Subversion. In a fresh directory:</li>
+</ol>
+<pre><code>svn co https://dist.apache.org/repos/dist/dev/carbondata
+</code></pre>
+<ol start="2">
+<li>Make a directory for the new release:</li>
+</ol>
+<pre><code>mkdir x.x.x
+</code></pre>
+<ol start="3">
+<li>Copy the CarbonData source distribution, hash, and GPG signature:</li>
+</ol>
+<pre><code>cp apache-carbondata-x.x.x-source-release.zip x.x.x
+</code></pre>
+<ol start="4">
+<li>Add and commit the files:</li>
+</ol>
+<pre><code>svn add x.x.x
+svn commit
+</code></pre>
+<ol start="5">
+<li>Verify the files are <a href="https://dist.apache.org/repos/dist/dev/carbondata" target=_blank rel="nofollow">present</a>.</li>
+</ol>
+<h3>
+<a id="propose-a-pull-request-for-website-updates" class="anchor" href="#propose-a-pull-request-for-website-updates" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Propose a pull request for website updates</h3>
+<p>The final step of building a release candidate is to propose a website pull request.</p>
+<p>This pull request should update the following page with the new release:</p>
+<ul>
+<li><code>src/main/webapp/index.html</code></li>
+<li><code>src/main/webapp/docs/latest/mainpage.html</code></li>
+</ul>
+<p><em>Checklist to proceed to the next step:</em></p>
+<ol>
+<li>Maven artifacts deployed to the staging repository of
+<a href="https://repository.apache.org" target=_blank rel="nofollow">repository.apache.org</a>
+</li>
+<li>Source distribution deployed to the dev repository of
+<a href="https://dist.apache.org/repos/dist/dev/carbondata/" target=_blank rel="nofollow">dist.apache.org</a>
+</li>
+<li>Website pull request to list the release.</li>
+</ol>
+<h2>
+<a id="vote-on-the-release-candidate" class="anchor" href="#vote-on-the-release-candidate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Vote on the release candidate</h2>
+<p>Once you have built and individually reviewed the release candidate, please share it for the
+community-wide review. Please review foundation-wide <a href="http://www.apache.org/foundation/voting.html" target=_blank rel="nofollow">voting guidelines</a>
+for more information.</p>
+<p>Start the review-and-vote thread on the dev@ mailing list. Here's an email template; please
+adjust as you see fit:</p>
+<pre><code>From: Release Manager
+To: dev@carbondata.apache.org
+Subject: [VOTE] Apache CarbonData Release x.x.x
+
+Hi everyone,
+Please review and vote on the release candidate for the version x.x.x, as follows:
+
+[ ] +1, Approve the release
+[ ] -1, Do not approve the release (please provide specific comments)
+
+The complete staging area is available for your review, which includes:
+* JIRA release notes [1],
+* the official Apache source release to be deployed to dist.apache.org [2], which is signed with the key with fingerprint FFFFFFFF [3],
+* all artifacts to be deployed to the Maven Central Repository [4],
+* source code tag "x.x.x" [5],
+* website pull request listing the release [6].
+
+The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.
+
+Thanks,
+Release Manager
+
+[1] link
+[2] link
+[3] https://dist.apache.org/repos/dist/dist/carbondata/KEYS
+[4] link
+[5] link
+[6] link
+</code></pre>
+<p>If there are any issues found in the release candidate, reply on the vote thread to cancel the vote.
+There?s no need to wait 72 hours. Proceed to the <code>Cancel a Release (Fix Issues)</code> step below and
+address the problem.
+However, some issues don?t require cancellation.
+For example, if an issue is found in the website pull request, just correct it on the spot and the
+vote can continue as-is.</p>
+<p>If there are no issues, reply on the vote thread to close the voting. Then, tally the votes in a
+separate email. Here?s an email template; please adjust as you see fit.</p>
+<pre><code>From: Release Manager
+To: dev@carbondata.apache.org
+Subject: [RESULT][VOTE] Apache CarbonData Release x.x.x
+
+I'm happy to announce that we have unanimously approved this release.
+
+There are XXX approving votes, XXX of which are binding:
+* approver 1
+* approver 2
+* approver 3
+* approver 4
+
+There are no disapproving votes.
+
+Thanks everyone!
+</code></pre>
+<p>While in incubation, the Apache Incubator PMC must also vote on each release, using the same
+process as above. Start the review and vote thread on the <code>general@incubator.apache.org</code> list.</p>
+<p><em>Checklist to proceed to the final step:</em></p>
+<ol>
+<li>Community votes to release the proposed release</li>
+<li>While in incubation, Apache Incubator PMC votes to release the proposed release</li>
+</ol>
+<h2>
+<a id="cancel-a-release-fix-issues" class="anchor" href="#cancel-a-release-fix-issues" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Cancel a Release (Fix Issues)</h2>
+<p>Any issue identified during the community review and vote should be fixed in this step.</p>
+<p>To fully cancel a vote:</p>
+<ul>
+<li>Cancel the current release and verify the version is back to the correct SNAPSHOT:</li>
+</ul>
+<pre><code>mvn release:cancel
+</code></pre>
+<ul>
+<li>Drop the release tag:</li>
+</ul>
+<pre><code>git tag -d x.x.x
+git push --delete apache x.x.x
+</code></pre>
+<ul>
+<li>Drop the staging repository on Nexus (<a href="https://repository.apache.org" target=_blank rel="nofollow">repository.apache.org</a>)</li>
+</ul>
+<p>Verify the version is back to the correct SNAPSHOT.</p>
+<p>Code changes should be proposed as standard pull requests and merged.</p>
+<p>Once all issues have been resolved, you should go back and build a new release candidate with
+these changes.</p>
+<h2>
+<a id="finalize-the-release" class="anchor" href="#finalize-the-release" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Finalize the release</h2>
+<p>Once the release candidate has been reviewed and approved by the community, the release should be
+finalized. This involves the final deployment of the release to the release repositories,
+merging the website changes, and announce the release.</p>
+<h3>
+<a id="deploy-artifacts-to-maven-central-repository" class="anchor" href="#deploy-artifacts-to-maven-central-repository" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploy artifacts to Maven Central repository</h3>
+<p>On Nexus, release the staged artifacts to Maven Central repository. In the <code>Staging Repositories</code>
+section, find the relevant release candidate <code>orgapachecarbondata-XXX</code> entry and click <code>Release</code>.</p>
+<h3>
+<a id="deploy-source-release-to-distapacheorg" class="anchor" href="#deploy-source-release-to-distapacheorg" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploy source release to dist.apache.org</h3>
+<p>Copy the source release from the <code>dev</code> repository to <code>release</code> repository at <code>dist.apache.org</code>
+using Subversion.</p>
+<h3>
+<a id="merge-website-pull-request" class="anchor" href="#merge-website-pull-request" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Merge website pull request</h3>
+<p>Merge the website pull request to list the release created earlier.</p>
+<h3>
+<a id="mark-the-version-as-released-in-jira" class="anchor" href="#mark-the-version-as-released-in-jira" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Mark the version as released in Jira</h3>
+<p>In Jira, inside <a href="https://issues.apache.org/jira/plugins/servlet/project-config/CARBONDATA/versions" target=_blank rel="nofollow">version management</a>
+, hover over the current release and a settings menu will appear. Click <code>Release</code>, and select
+today's state.</p>
+<p><em>Checklist to proceed to the next step:</em></p>
+<ol>
+<li>Maven artifacts released and indexed in the
+<a href="https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.carbondata%22" rel="nofollow">Maven Central repository</a>
+</li>
+<li>Source distribution available in the release repository of
+<a href="https://dist.apache.org/repos/dist/release/carbondata/" target=_blank rel="nofollow">dist.apache.org</a>
+</li>
+<li>Website pull request to list the release merged</li>
+<li>Release version finalized in Jira</li>
+</ol>
+<h2>
+<a id="promote-the-release" class="anchor" href="#promote-the-release" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Promote the release</h2>
+<p>Once the release has been finalized, the last step of the process is to promote the release
+within the project and beyond.</p>
+<h3>
+<a id="apache-mailing-lists" class="anchor" href="#apache-mailing-lists" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Apache mailing lists</h3>
+<p>Announce on the dev@ mailing list that the release has been finished.</p>
+<p>Announce on the user@ mailing list that the release is available, listing major improvements and
+contributions.</p>
+<p>While in incubation, announce the release on the Incubator's general@ mailing list.</p>
+<p><em>Checklist to declare the process completed:</em></p>
+<ol>
+<li>Release announced on the user@ mailing list.</li>
+<li>Release announced on the Incubator's general@ mailing list.</li>
+<li>Completion declared on the dev@ mailing list.</li>
+</ol>
+</div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/content/timeseries-datamap-guide.html
----------------------------------------------------------------------
diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html
new file mode 100644
index 0000000..ce36e72
--- /dev/null
+++ b/content/timeseries-datamap-guide.html
@@ -0,0 +1,322 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"
+ target="_blank">Apache CarbonData 1.3.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"
+ target="_blank">Apache CarbonData 1.2.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"
+ target="_blank">Apache CarbonData 1.1.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"
+ target="_blank">Apache CarbonData 1.1.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"
+ target="_blank">Apache CarbonData 1.0.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"
+ target="_blank">Apache CarbonData 0.2.0</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"
+ target="_blank">Apache CarbonData 0.1.1</a></li>
+ <li>
+ <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"
+ target="_blank">Apache CarbonData 0.1.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="mainpage.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="row">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="carbondata-timeseries-datamap" class="anchor" href="#carbondata-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Timeseries DataMap</h1>
+<ul>
+<li><a href="#timeseries-datamap-intoduction-(alpha-feature-in-1.3.0)">Timeseries DataMap</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
+</ul>
+<h2>
+<a id="timeseries-datamap-intoduction-alpha-feature-in-130" class="anchor" href="#timeseries-datamap-intoduction-alpha-feature-in-130" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap Intoduction (Alpha feature in 1.3.0)</h2>
+<p>Timeseries DataMap a pre-aggregate table implementation based on 'preaggregate' DataMap.
+Difference is that Timerseries DataMap has built-in understanding of time hierarchy and
+levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension
+for query.</p>
+<p>The data loading, querying, compaction command and its behavior is the same as preaggregate DataMap.
+Please refer to <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>Pre-aggregate DataMap</a>
+for more information.</p>
+<p>To use this datamap, user can create multiple timeseries datamap on the main table which has
+a <em>event_time</em> column, one datamap for one time granularity. Then Carbondata can do automatic
+roll-up for queries on the main table.</p>
+<p>For example, below statement effectively create multiple pre-aggregate tables on main table called
+<strong>timeseries</strong></p>
+<pre><code>CREATE DATAMAP agg_year
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'year_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_month
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'month_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_day
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_sales_hour
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granualrity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>For querying timeseries data, Carbondata has builtin support for following time related UDF
+to enable automatically roll-up to the desired aggregation level</p>
+<pre><code>timeseries(timeseries column name, 'aggregation level')
+</code></pre>
+<pre><code>SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales GROUP BY timeseries(order_time,
+'hour')
+</code></pre>
+<p>It is <strong>not necessary</strong> to create pre-aggregate tables for each granularity unless required for
+query. Carbondata can roll-up the data and fetch it.</p>
+<p>For Example: For main table <strong>sales</strong> , if following timeseries datamaps were created for day
+level and hour level pre-aggregate</p>
+<pre><code> CREATE DATAMAP agg_day
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+ CREATE DATAMAP agg_sales_hour
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granualrity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>Queries like below will be rolled-up and hit the timeseries datamaps</p>
+<pre><code>Select timeseries(order_time, 'month'), sum(quantity) from sales group by timeseries(order_time,
+ 'month')
+
+Select timeseries(order_time, 'year'), sum(quantity) from sales group by timeseries(order_time,
+ 'year')
+</code></pre>
+<p>NOTE (<b>RESTRICTION</b>):</p>
+<ul>
+<li>Only value of 1 is supported for hierarchy levels. Other hierarchy levels will be supported in
+the future CarbonData release.</li>
+<li>timeseries datamap for the desired levels needs to be created one after the other</li>
+<li>timeseries datamaps created for each level needs to be dropped separately</li>
+</ul>
+<h2>
+<a id="compacting-timeseries-datamp" class="anchor" href="#compacting-timeseries-datamp" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting timeseries datamp</h2>
+<p>Refer to Compaction section in <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<h2>
+<a id="data-management-on-timeseries-datamap" class="anchor" href="#data-management-on-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management on timeseries datamap</h2>
+<p>Refer to Data Management section in <a href="https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.html" target=_blank>preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/resources/application.conf
----------------------------------------------------------------------
diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf
index e20ebe4..5fe9837 100644
--- a/src/main/resources/application.conf
+++ b/src/main/resources/application.conf
@@ -9,6 +9,12 @@ fileList=["configuration-parameters",
"troubleshooting",
"useful-tips-on-carbondata"
]
+dataMapFileList=[
+ "preaggregate-datamap-guide",
+ "timeseries-datamap-guide"]
+
+dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/datamap/"
+
fileListToRetain=["quick-start-guide",
"installation-guide"
]
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a4bd7be6/src/main/scala/MDFileConverter.scala
----------------------------------------------------------------------
diff --git a/src/main/scala/MDFileConverter.scala b/src/main/scala/MDFileConverter.scala
index 59cb7ad..79d1d27 100644
--- a/src/main/scala/MDFileConverter.scala
+++ b/src/main/scala/MDFileConverter.scala
@@ -1,13 +1,13 @@
import com.google.inject.Inject
import org.slf4j.{Logger, LoggerFactory}
import services.{ConfService, DataService, FileService}
-import java.nio.file.{Files, Path, Paths}
import scala.util.matching.Regex
class MDFileConverter @Inject()(fileService: FileService, confService: ConfService, dataService: DataService) {
val logger: Logger = LoggerFactory.getLogger(classOf[MDFileConverter])
val url: String = confService.readString("apiUrl")
+ val dataMapFileUrl: String = confService.readString("dataMapFilesUrl")
val inputFileExtension: String = ".md"
val outputFileExtension: String = ".html"
val headerContent: String = fileService.readFromFile(confService.readString("headerPath"))
@@ -15,45 +15,29 @@ class MDFileConverter @Inject()(fileService: FileService, confService: ConfServi
val location: String = confService.readString("outputFileLocation")
val fileReadObject: MdFileHandler = new MdFileHandler(confService, fileService)
val failMessage: String = "failure"
+ val imageFileList: List[String] = confService.readListOfString("imagesFilesList")
/**
- * reads list of files , converts file extension to output file extension and writes file to the location
+ * reads list of files , converts file extension to output file extension and writes file to
+ * the location
*
* @return status of each file i.e. success or failure
*/
- def convertToHtml(status: Boolean): String = {
+ def convertToHtml(dirStatus: Boolean): String = {
val listOfFiles: List[String] = confService.readListOfString("fileList")
- val imageFileList: List[String] =confService.readListOfString("imagesFilesList")
+ val listOfDataMapFiles: List[String] = confService.readListOfString("dataMapFileList")
val statusList: List[String] = listOfFiles.map { file =>
- val fileURLContent: String = dataService.dataOnGetRequest(url + file + inputFileExtension)
- val getFileData: Option[String] = dataService.dataOnPostRequest(fileURLContent)
- getFileData match {
- case Some(data: String) => val fileData = fileReadObject.convertMdExtensions(data)
- logger.info(s"Begin writing [ $file outputFileExtension ] at $location")
- val statusHtmlFile = fileService.writeToFile(location + file + outputFileExtension, headerContent + fileData + footerContent)
- saveMdFilesForPDF(status, fileURLContent, file)
- if (imageFileList.contains(file)) {
- saveMdFilesForPDF(status, changeImageLink(fileURLContent), file)
- }
- else {
- saveMdFilesForPDF(status, fileURLContent, file)
- }
- if (statusHtmlFile) {
- logger.info(s"Successfully written [ $file $outputFileExtension ] at $location")
- "Success"
- }
- else {
- failMessage
- }
- case None => logger.error(s"$file Conversion failed !")
- failMessage
- }
+ convertToHtmlFromPath(url, location + file + outputFileExtension, dirStatus, file)
+ }
+
+ val dataFileStatusList: List[String] = listOfDataMapFiles.map { file =>
+ convertToHtmlFromPath(dataMapFileUrl, location + file + outputFileExtension, dirStatus, file)
}
fileReadObject.convertReadMeExtensions()
logger.info("Restored back the ReadMe.md file extension in Installation Guide")
- if (statusList.contains("Failure")) {
+ if ((statusList::dataFileStatusList).contains("Failure")) {
"[ERROR]: Some Files Failed To Convert"
}
else {
@@ -89,5 +73,36 @@ class MDFileConverter @Inject()(fileService: FileService, confService: ConfServi
val contentAfterModifyImageLink: String = modifyImagePattern replaceAllIn(content, "../../src/site/images/"+"$2"+"")
contentAfterModifyImageLink
}
-}
+ private def convertToHtmlFromPath(urlPath: String,
+ outputPath: String,
+ dirStatus: Boolean,
+ fileName: String): String = {
+ val fileURLContent: String = dataService
+ .dataOnGetRequest(urlPath + fileName + inputFileExtension)
+ val getFileData: Option[String] = dataService.dataOnPostRequest(fileURLContent)
+ getFileData match {
+ case Some(data: String) => val fileData = fileReadObject.convertMdExtensions(data)
+ logger.info(s"Begin writing [ $fileName outputFileExtension ] at $location")
+ val statusHtmlFile = fileService
+ .writeToFile(outputPath, headerContent + fileData + footerContent)
+ saveMdFilesForPDF(dirStatus, fileURLContent, fileName)
+ if (imageFileList.contains(fileName)) {
+ saveMdFilesForPDF(dirStatus, changeImageLink(fileURLContent), fileName)
+ }
+ else {
+ saveMdFilesForPDF(dirStatus, fileURLContent, fileName)
+ }
+ if (statusHtmlFile) {
+ logger.info(s"Successfully written [ $fileName $outputFileExtension ] at $location")
+ "Success"
+ }
+ else {
+ failMessage
+ }
+ case None => logger.error(s"$fileName Conversion failed !")
+ failMessage
+ }
+ }
+
+}
[3/3] carbondata-site git commit: added preaggregate-datamap-guide.md
and timeseries-datamap-guide.md files for datamap
Posted by ch...@apache.org.
added preaggregate-datamap-guide.md and timeseries-datamap-guide.md files for datamap
Project: http://git-wip-us.apache.org/repos/asf/carbondata-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata-site/commit/e4ea53ff
Tree: http://git-wip-us.apache.org/repos/asf/carbondata-site/tree/e4ea53ff
Diff: http://git-wip-us.apache.org/repos/asf/carbondata-site/diff/e4ea53ff
Branch: refs/heads/asf-site
Commit: e4ea53ff1bd9bae4a8bb738ccfdd849ef1a4712e
Parents: b0888c1 a4bd7be
Author: chenliang613 <ch...@huawei.com>
Authored: Wed Mar 7 20:20:47 2018 +0800
Committer: chenliang613 <ch...@huawei.com>
Committed: Wed Mar 7 20:20:47 2018 +0800
----------------------------------------------------------------------
content/WEB-INF/classes/MDFileConverter.class | Bin 8964 -> 10321 bytes
content/WEB-INF/classes/application.conf | 6 +
content/index.html | 2 +-
content/mainpage.html | 5 +-
content/pdf/CarbonData Documentation.pdf | Bin 224247 -> 239761 bytes
content/preaggregate-datamap-guide.html | 440 ++++++++++++++++
content/release-guide.html | 526 +++++++++++++++++++
content/timeseries-datamap-guide.html | 322 ++++++++++++
src/main/resources/application.conf | 6 +
src/main/scala/MDFileConverter.scala | 73 ++-
src/main/webapp/index.html | 2 +-
src/main/webapp/mainpage.html | 5 +-
.../webapp/pdf/CarbonData Documentation.pdf | Bin 224247 -> 239761 bytes
src/main/webapp/preaggregate-datamap-guide.html | 440 ++++++++++++++++
src/main/webapp/timeseries-datamap-guide.html | 322 ++++++++++++
src/site/markdown/preaggregate-datamap-guide.md | 252 +++++++++
src/site/markdown/timeseries-datamap-guide.md | 146 +++++
src/site/pdf.xml | 2 +
18 files changed, 2514 insertions(+), 35 deletions(-)
----------------------------------------------------------------------