You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ra...@apache.org on 2020/05/31 13:44:23 UTC
[carbondata-site] branch asf-site updated: Added 2.0.0 version
information (#74)
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/carbondata-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 03659a0 Added 2.0.0 version information (#74)
03659a0 is described below
commit 03659a003d2323c54103cffba34c53ca03f99d8a
Author: sraghunandan <ca...@gmail.com>
AuthorDate: Sun May 31 19:14:16 2020 +0530
Added 2.0.0 version information (#74)
This closes #74
---
content/CSDK-guide.html | 687 ---------------------
content/WEB-INF/classes/META-INF/NOTICE | 2 +-
content/WEB-INF/classes/application.conf | 24 +-
content/alluxio-guide.html | 37 +-
...map-guide.html => bloomfilter-index-guide.html} | 155 ++---
content/carbon-as-spark-datasource-guide.html | 29 +-
content/configuration-parameters.html | 270 ++++----
content/csdk-guide.html | 30 +-
content/ddl-of-carbondata.html | 225 +++----
content/dml-of-carbondata.html | 272 ++++----
content/documentation.html | 31 +-
content/faq.html | 100 ++-
content/file-structure-of-carbondata.html | 28 +-
content/flink-integration-guide.html | 494 +++++++++++++++
content/hive-guide.html | 52 +-
.../how-to-contribute-to-apache-carbondata.html | 27 +-
...loper-guide.html => index-developer-guide.html} | 50 +-
.../index-management.html | 169 +++--
content/index-server.html | 124 ++--
content/index.html | 22 +-
content/introduction.html | 45 +-
content/language-manual.html | 38 +-
...-datamap-guide.html => lucene-index-guide.html} | 148 ++---
content/{mv-datamap-guide.html => mv-guide.html} | 419 ++++++++-----
content/performance-tuning.html | 49 +-
content/preaggregate-datamap-guide.html | 519 ----------------
.../prestodb-guide.html | 52 +-
.../prestosql-guide.html | 90 +--
content/quick-start-guide.html | 390 ++++++------
content/release-guide.html | 27 +-
content/s3-guide.html | 27 +-
.../scd-and-cdc-guide.html | 137 ++--
content/sdk-guide.html | 54 +-
.../secondary-index-guide.html | 289 ++++-----
content/security.html | 6 +-
content/segment-management-on-carbondata.html | 119 +++-
.../spatial-index-guide.html | 238 +++----
content/streaming-guide.html | 49 +-
content/supported-data-types-in-carbondata.html | 27 +-
content/timeseries-datamap-guide.html | 380 ------------
content/usecases.html | 45 +-
content/videogallery.html | 6 +-
src/main/resources/application.conf | 24 +-
src/main/scala/html/header.html | 27 +-
src/main/scala/scripts/bloomfilter-datamap-guide | 11 -
...eries-datamap-guide => bloomfilter-index-guide} | 0
...map-developer-guide => flink-integration-guide} | 0
...tamap-developer-guide => index-developer-guide} | 0
...preaggregate-datamap-guide => index-management} | 0
.../{lucene-datamap-guide => lucene-index-guide} | 0
src/main/scala/scripts/mv-datamap-guide | 11 -
.../scripts/{datamap-developer-guide => mv-guide} | 0
.../scala/scripts/{presto-guide => prestodb-guide} | 0
.../scripts/{presto-guide => prestosql-guide} | 0
.../scripts/{presto-guide => scd-and-cdc-guide} | 0
.../{datamap-management => secondary-index-guide} | 0
.../scripts/{presto-guide => spatial-index-guide} | 0
src/main/webapp/alluxio-guide.html | 37 +-
...map-guide.html => bloomfilter-index-guide.html} | 155 ++---
.../webapp/carbon-as-spark-datasource-guide.html | 29 +-
src/main/webapp/configuration-parameters.html | 270 ++++----
src/main/webapp/csdk-guide.html | 30 +-
src/main/webapp/ddl-of-carbondata.html | 225 +++----
src/main/webapp/dml-of-carbondata.html | 272 ++++----
src/main/webapp/documentation.html | 31 +-
src/main/webapp/faq.html | 100 ++-
src/main/webapp/file-structure-of-carbondata.html | 28 +-
src/main/webapp/flink-integration-guide.html | 494 +++++++++++++++
src/main/webapp/hive-guide.html | 52 +-
.../how-to-contribute-to-apache-carbondata.html | 27 +-
...loper-guide.html => index-developer-guide.html} | 50 +-
.../main/webapp/index-management.html | 169 +++--
src/main/webapp/index-server.html | 124 ++--
src/main/webapp/index.html | 6 +-
src/main/webapp/introduction.html | 45 +-
src/main/webapp/language-manual.html | 38 +-
...-datamap-guide.html => lucene-index-guide.html} | 148 ++---
.../main/webapp/mv-guide.html | 419 ++++++++-----
src/main/webapp/performance-tuning.html | 49 +-
src/main/webapp/preaggregate-datamap-guide.html | 519 ----------------
.../main/webapp/prestodb-guide.html | 52 +-
.../{presto-guide.html => prestosql-guide.html} | 90 +--
src/main/webapp/quick-start-guide.html | 390 ++++++------
src/main/webapp/release-guide.html | 27 +-
src/main/webapp/s3-guide.html | 27 +-
.../{s3-guide.html => scd-and-cdc-guide.html} | 137 ++--
src/main/webapp/sdk-guide.html | 54 +-
.../main/webapp/secondary-index-guide.html | 289 ++++-----
src/main/webapp/security.html | 6 +-
.../webapp/segment-management-on-carbondata.html | 119 +++-
...datamap-guide.html => spatial-index-guide.html} | 238 +++----
src/main/webapp/streaming-guide.html | 49 +-
.../webapp/supported-data-types-in-carbondata.html | 27 +-
src/main/webapp/timeseries-datamap-guide.html | 380 ------------
src/main/webapp/usecases.html | 45 +-
src/main/webapp/videogallery.html | 6 +-
src/site/markdown/CSDK-guide.md | 3 +-
src/site/markdown/alluxio-guide.md | 10 +-
src/site/markdown/bloomfilter-datamap-guide.md | 148 -----
src/site/markdown/bloomfilter-index-guide.md | 150 +++++
.../markdown/carbon-as-spark-datasource-guide.md | 2 +-
src/site/markdown/configuration-parameters.md | 91 +--
src/site/markdown/datamap-developer-guide.md | 36 --
src/site/markdown/datamap-management.md | 152 -----
src/site/markdown/ddl-of-carbondata.md | 203 ++----
src/site/markdown/dml-of-carbondata.md | 249 ++++----
src/site/markdown/documentation.md | 2 +-
src/site/markdown/faq.md | 93 +--
src/site/markdown/file-structure-of-carbondata.md | 2 -
src/site/markdown/flink-integration-guide.md | 225 +++++++
src/site/markdown/hive-guide.md | 25 +-
src/site/markdown/index-developer-guide.md | 35 ++
src/site/markdown/index-management.md | 127 ++++
src/site/markdown/index-server.md | 65 +-
src/site/markdown/introduction.md | 16 +-
src/site/markdown/language-manual.md | 11 +-
...cene-datamap-guide.md => lucene-index-guide.md} | 123 ++--
src/site/markdown/mv-datamap-guide.md | 210 -------
src/site/markdown/mv-guide.md | 341 ++++++++++
src/site/markdown/performance-tuning.md | 16 +-
src/site/markdown/preaggregate-datamap-guide.md | 275 ---------
.../{presto-guide.md => prestodb-guide.md} | 25 +-
.../{presto-guide.md => prestosql-guide.md} | 63 +-
src/site/markdown/quick-start-guide.md | 302 ++++-----
src/site/markdown/scd-and-cdc-guide.md | 62 ++
src/site/markdown/sdk-guide.md | 27 +-
src/site/markdown/secondary-index-guide.md | 191 ++++++
.../markdown/segment-management-on-carbondata.md | 66 +-
src/site/markdown/spatial-index-guide.md | 97 +++
src/site/markdown/streaming-guide.md | 10 +-
src/site/markdown/timeseries-datamap-guide.md | 154 -----
src/site/markdown/usecases.md | 8 +-
132 files changed, 7066 insertions(+), 7857 deletions(-)
diff --git a/content/CSDK-guide.html b/content/CSDK-guide.html
deleted file mode 100644
index d775939..0000000
--- a/content/CSDK-guide.html
+++ /dev/null
@@ -1,687 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
- <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
- <title>CarbonData</title>
- <style>
-
- </style>
- <!-- Bootstrap -->
-
- <link rel="stylesheet" href="css/bootstrap.min.css">
- <link href="css/style.css" rel="stylesheet">
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- <script src="js/jquery.min.js"></script>
- <script src="js/bootstrap.min.js"></script>
- <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
-
-
-</head>
-<body>
-<header>
- <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
- <div class="container">
- <div class="navbar-header">
- <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
- class="navbar-toggle collapsed" type="button">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a href="index.html" class="logo">
- <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
- </a>
- </div>
- <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
- <ul class="nav navbar-nav navbar-right navlist-custom">
- <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
- </li>
- <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false"> Download <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
- target="_blank">Apache CarbonData 1.6.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.0/"
- target="_blank">Apache CarbonData 1.6.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.4/"
- target="_blank">Apache CarbonData 1.5.4</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.3/"
- target="_blank">Apache CarbonData 1.5.3</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.2/"
- target="_blank">Apache CarbonData 1.5.2</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"
- target="_blank">Apache CarbonData 1.5.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
- target="_blank">Apache CarbonData 1.5.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
- target="_blank">Release Archive</a></li>
- </ul>
- </li>
- <li><a href="documentation.html" class="active">Documentation</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false">Community <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
- target="_blank">Contributing to CarbonData</a></li>
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
- target="_blank">Release Guide</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
- target="_blank">Project PMC and Committers</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
- target="_blank">CarbonData Meetups</a></li>
- <li><a href="security.html">Apache CarbonData Security</a></li>
- <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
- Jira</a></li>
- <li><a href="videogallery.html">CarbonData Videos </a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li class="dropdown">
- <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li>
- <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
-
- </li>
-
- </ul>
- </div><!--/.nav-collapse -->
- <div id="search-box">
- <form method="get" action="http://www.google.com/search" target="_blank">
- <div class="search-block">
- <table border="0" cellpadding="0" width="100%">
- <tr>
- <td style="width:80%">
- <input type="text" name="q" size=" 5" maxlength="255" value=""
- class="search-input" placeholder="Search...." required/>
- </td>
- <td style="width:20%">
- <input type="submit" value="Search"/></td>
- </tr>
- <tr>
- <td align="left" style="font-size:75%" colspan="2">
- <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
- <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
- </td>
- </tr>
- </table>
- </div>
- </form>
- </div>
- </div>
- </nav>
-</header> <!-- end Header part -->
-
-<div class="fixed-padding"></div> <!-- top padding with fixde header -->
-
-<section><!-- Dashboard nav -->
- <div class="container-fluid q">
- <div class="col-sm-12 col-md-12 maindashboard">
- <div class="verticalnavbar">
- <nav class="b-sticky-nav">
- <div class="nav-scroller">
- <div class="nav__inner">
- <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
- <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
- <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
- <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
- <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
- <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
- <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
- <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
- <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
- <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
- <a class="nav__item nav__sub__item" href="./csdk-guide.html">C++ SDK</a>
- </div>
-
- <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
- <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
- <a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
- <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
- <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
- <a class="b-nav__security nav__item" href="./security.html">Security</a>
- <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
- </div>
- </div>
- <div class="navindicator">
- <div class="b-nav__intro navindicator__item"></div>
- <div class="b-nav__quickstart navindicator__item"></div>
- <div class="b-nav__uses navindicator__item"></div>
- <div class="b-nav__docs navindicator__item"></div>
- <div class="b-nav__datamap navindicator__item"></div>
- <div class="b-nav__api navindicator__item"></div>
- <div class="b-nav__perf navindicator__item"></div>
- <div class="b-nav__s3 navindicator__item"></div>
- <div class="b-nav__indexserver navindicator__item"></div>
- <div class="b-nav__faq navindicator__item"></div>
- <div class="b-nav__contri navindicator__item"></div>
- <div class="b-nav__security navindicator__item"></div>
- </div>
- </nav>
- </div>
- <div class="mdcontent">
- <section>
- <div style="padding:10px 15px;">
- <div id="viewpage" name="viewpage">
- <div class="row">
- <div class="col-sm-12 col-md-12">
- <div>
-<h1>
-<a id="c-sdk-guide" class="anchor" href="#c-sdk-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>C++ SDK Guide</h1>
-<p>CarbonData C++ SDK provides C++ interface to write and read carbon file.
-C++ SDK use JNI to invoke java SDK in C++ code.</p>
-<h1>
-<a id="c-sdk-reader" class="anchor" href="#c-sdk-reader" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>C++ SDK Reader</h1>
-<p>This C++ SDK reader reads CarbonData file and carbonindex file at a given path.
-External client can make use of this reader to read CarbonData files in C++
-code and without CarbonSession.</p>
-<p>In the carbon jars package, there exist a carbondata-sdk.jar,
-including SDK reader for C++ SDK.</p>
-<h2>
-<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
-<p>Please find example code at <a href="https://github.com/apache/carbondata/blob/master/store/CSDK/test/main.cpp" target=_blank>main.cpp</a> of CSDK module</p>
-<p>When users use C++ to read carbon files, users should init JVM first. Then users create
-carbon reader and read data.There are some example code of read data from local disk<br>
-and read data from S3 at main.cpp of CSDK module. Finally, users need to
-release the memory and destroy JVM.</p>
-<p>C++ SDK support read batch row. User can set batch by using withBatch(int batch) before build, and read batch by using readNextBatchRow().</p>
-<h2>
-<a id="api-list" class="anchor" href="#api-list" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2>
-<h3>
-<a id="carbonreader" class="anchor" href="#carbonreader" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonReader</h3>
-<pre><code>/**
- * Create a CarbonReaderBuilder object for building carbonReader,
- * CarbonReaderBuilder object can configure different parameter
- *
- * @param env JNIEnv
- * @param path data store path
- * @param tableName table name
- * @return CarbonReaderBuilder object
- */
-jobject builder(JNIEnv *env, char *path, char *tableName);
-</code></pre>
-<pre><code>/**
- * Create a CarbonReaderBuilder object for building carbonReader,
- * CarbonReaderBuilder object can configure different parameter
- *
- * @param env JNIEnv
- * @param path data store path
- *
- */
-void builder(JNIEnv *env, char *path);
-</code></pre>
-<pre><code>/**
- * Configure the projection column names of carbon reader
- *
- * @param argc argument counter
- * @param argv argument vector
- * @return CarbonReaderBuilder object
- */
-jobject projection(int argc, char *argv[]);
-</code></pre>
-<pre><code>/**
- * Build carbon reader with argument vector
- * it supports multiple parameters
- * like: key=value
- * for example: fs.s3a.access.key=XXXX, XXXX is user's access key value
- *
- * @param argc argument counter
- * @param argv argument vector
- * @return CarbonReaderBuilder object
- *
- */
-jobject withHadoopConf(int argc, char *argv[]);
-</code></pre>
-<pre><code>/**
- * Sets the batch size of records to read
- *
- * @param batch batch size
- * @return CarbonReaderBuilder object
- */
-void withBatch(int batch);
-</code></pre>
-<pre><code>/**
- * Configure Row Record Reader for reading.
- */
-void withRowRecordReader();
-</code></pre>
-<pre><code>/**
- * Build carbonReader object for reading data
- * it supports read data from load disk
- *
- * @return carbonReader object
- */
-jobject build();
-</code></pre>
-<pre><code>/**
- * Whether it has next row data
- *
- * @return boolean value, if it has next row, return true. if it hasn't next row, return false.
- */
-jboolean hasNext();
-</code></pre>
-<pre><code>/**
- * Read next carbonRow from data
- * @return carbonRow object of one row
- */
-jobject readNextRow();
-</code></pre>
-<pre><code>/**
- * Read Next Batch Row
- *
- * @return rows
- */
-jobjectArray readNextBatchRow();
-</code></pre>
-<pre><code>/**
- * Close the carbon reader
- *
- * @return boolean value
- */
-jboolean close();
-</code></pre>
-<h1>
-<a id="c-sdk-writer" class="anchor" href="#c-sdk-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>C++ SDK Writer</h1>
-<p>This C++ SDK writer writes CarbonData file and carbonindex file at a given path.
-External client can make use of this writer to write CarbonData files in C++
-code and without CarbonSession. C++ SDK already supports S3 and local disk.</p>
-<p>In the carbon jars package, there exist a carbondata-sdk.jar,
-including SDK writer for C++ SDK.</p>
-<h2>
-<a id="quick-example-1" class="anchor" href="#quick-example-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
-<p>Please find example code at <a href="https://github.com/apache/carbondata/blob/master/store/CSDK/test/main.cpp" target=_blank>main.cpp</a> of CSDK module</p>
-<p>When users use C++ to write carbon files, users should init JVM first. Then users create
-carbon writer and write data.There are some example code of write data to local disk<br>
-and write data to S3 at main.cpp of CSDK module. Finally, users need to
-release the memory and destroy JVM.</p>
-<h2>
-<a id="api-list-1" class="anchor" href="#api-list-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2>
-<h3>
-<a id="carbonwriter" class="anchor" href="#carbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonWriter</h3>
-<pre><code>/**
- * Create a CarbonWriterBuilder object for building carbonWriter,
- * CarbonWriterBuilder object can configure different parameter
- *
- * @param env JNIEnv
- * @return CarbonWriterBuilder object
- */
-void builder(JNIEnv *env);
-</code></pre>
-<pre><code>/**
- * Sets the output path of the writer builder
- *
- * @param path is the absolute path where output files are written
- * This method must be called when building CarbonWriterBuilder
- * @return updated CarbonWriterBuilder
- */
-void outputPath(char *path);
-</code></pre>
-<pre><code>/**
- * Sets the list of columns that needs to be in sorted order
- *
- * @param argc argc argument counter, the number of projection column
- * @param argv argv is a string array of columns that needs to be sorted.
- * If it is null or by default all dimensions are selected for sorting
- * If it is empty array, no columns are sorted
- */
-void sortBy(int argc, char *argv[]);
-</code></pre>
-<pre><code>/**
- * Configure the schema with json style schema
- *
- * @param jsonSchema json style schema
- * @return updated CarbonWriterBuilder
- */
-void withCsvInput(char *jsonSchema);
-</code></pre>
-<pre><code>/**
- * Updates the hadoop configuration with the given key value
- *
- * @param key key word
- * @param value value
- * @return CarbonWriterBuilder object
- */
-void withHadoopConf(char *key, char *value);
-</code></pre>
-<pre><code>/**
- * To support the table properties for writer
- *
- * @param key properties key
- * @param value properties value
- */
-void withTableProperty(char *key, char *value);
-</code></pre>
-<pre><code>/**
- * To support the load options for C++ sdk writer
- *
- * @param options key,value pair of load options.
- * supported keys values are
- * a. bad_records_logger_enable -- true (write into separate logs), false
- * b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT
- * c. bad_record_path -- path
- * d. dateformat -- same as JAVA SimpleDateFormat
- * e. timestampformat -- same as JAVA SimpleDateFormat
- * f. complex_delimiter_level_1 -- value to Split the complexTypeData
- * g. complex_delimiter_level_2 -- value to Split the nested complexTypeData
- * h. quotechar
- * i. escapechar
- *
- * Default values are as follows.
- *
- * a. bad_records_logger_enable -- "false"
- * b. bad_records_action -- "FAIL"
- * c. bad_record_path -- ""
- * d. dateformat -- "" , uses from carbon.properties file
- * e. timestampformat -- "", uses from carbon.properties file
- * f. complex_delimiter_level_1 -- "$"
- * g. complex_delimiter_level_2 -- ":"
- * h. quotechar -- "\""
- * i. escapechar -- "\\"
- *
- * @return updated CarbonWriterBuilder
- */
-void withLoadOption(char *key, char *value);
-</code></pre>
-<pre><code>/**
- * Sets the taskNo for the writer. CSDKs concurrently running
- * will set taskNo in order to avoid conflicts in file's name during write.
- *
- * @param taskNo is the TaskNo user wants to specify.
- * by default it is system time in nano seconds.
- */
-void taskNo(long taskNo);
-</code></pre>
-<pre><code>/**
- * Set the timestamp in the carbondata and carbonindex index files
- *
- * @param timestamp is a timestamp to be used in the carbondata and carbonindex index files.
- * By default set to zero.
- * @return updated CarbonWriterBuilder
- */
-void uniqueIdentifier(long timestamp);
-</code></pre>
-<pre><code>/**
- * To make c++ sdk writer thread safe.
- *
- * @param numOfThreads should number of threads in which writer is called in multi-thread scenario
- * default C++ sdk writer is not thread safe.
- * can use one writer instance in one thread only.
- */
-void withThreadSafe(short numOfThreads) ;
-</code></pre>
-<pre><code>/**
- * To set the carbondata file size in MB between 1MB-2048MB
- *
- * @param blockSize is size in MB between 1MB to 2048 MB
- * default value is 1024 MB
- */
-void withBlockSize(int blockSize);
-</code></pre>
-<pre><code>/**
- * To set the blocklet size of CarbonData file
- *
- * @param blockletSize is blocklet size in MB
- * default value is 64 MB
- * @return updated CarbonWriterBuilder
- */
-void withBlockletSize(int blockletSize);
-</code></pre>
-<pre><code>/**
- * @param localDictionaryThreshold is localDictionaryThreshold, default is 10000
- * @return updated CarbonWriterBuilder
- */
-void localDictionaryThreshold(int localDictionaryThreshold);
-</code></pre>
-<pre><code>/**
- * @param enableLocalDictionary enable local dictionary, default is false
- * @return updated CarbonWriterBuilder
- */
-void enableLocalDictionary(bool enableLocalDictionary);
-</code></pre>
-<pre><code>/**
- * @param appName appName which is writing the carbondata files
- */
-void writtenBy(char *appName);
-</code></pre>
-<pre><code>/**
- * Build carbonWriter object for writing data
- * it support write data from load disk
- *
- * @return carbonWriter object
- */
-void build();
-</code></pre>
-<pre><code>/**
- * Write an object to the file, the format of the object depends on the
- * implementation.
- * Note: This API is not thread safe
- */
-void write(jobject obj);
-</code></pre>
-<pre><code>/**
- * close the carbon Writer
- */
-void close();
-</code></pre>
-<h3>
-<a id="carbonschemareader" class="anchor" href="#carbonschemareader" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonSchemaReader</h3>
-<pre><code>/**
- * Constructor with jni env
- *
- * @param env jni env
- */
-CarbonSchemaReader(JNIEnv *env);
-</code></pre>
-<pre><code>/**
- * Read schema from path,
- * path can be folder path, carbonindex file path, and carbondata file path
- * and will not check all files schema
- *
- * @param path file/folder path
- * @return schema
- */
-jobject readSchema(char *path);
-</code></pre>
-<pre><code>/**
- * Read schema from path,
- * path can be folder path, carbonindex file path, and carbondata file path
- * and user can decide whether check all files schema
- *
- * @param path carbon data path
- * @param validateSchema whether check all files schema
- * @return schema
- */
-jobject readSchema(char *path, bool validateSchema);
-</code></pre>
-<pre><code>/**
- * Read schema from path,
- * path can be folder path, carbonindex file path, and carbondata file path
- * and will not check all files schema
- *
- * @param path file/folder path
- * @param conf configuration support, can set s3a AK,SK,
- * end point and other conf with this
- * @return schema
- */
-jobject readSchema(char *path, Configuration conf);
-</code></pre>
-<pre><code>/**
- * Read schema from path,
- * path can be folder path, carbonindex file path, and carbondata file path
- * and user can decide whether check all files schema
- *
- * @param path carbon data path
- * @param validateSchema whether check all files schema
- * @param conf configuration support, can set s3a AK,SK,
- * end point and other conf with this
- * @return schema
- */
-jobject readSchema(char *path, bool validateSchema, Configuration conf);
-</code></pre>
-<h3>
-<a id="schema" class="anchor" href="#schema" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Schema</h3>
-<pre><code>/**
- * Constructor with jni env and carbon schema data
- *
- * @param env jni env
- * @param schema carbon schema data
- */
-Schema(JNIEnv *env, jobject schema);
-</code></pre>
-<pre><code>/**
- * Get fields length of schema
- *
- * @return fields length
- */
-int getFieldsLength();
-</code></pre>
-<pre><code>/**
- * Get field name by ordinal
- *
- * @param ordinal the data index of carbon schema
- * @return ordinal field name
- */
-char *getFieldName(int ordinal);
-</code></pre>
-<pre><code>/**
- * Get field data type name by ordinal
- *
- * @param ordinal the data index of carbon schema
- * @return ordinal field data type name
- */
-char *getFieldDataTypeName(int ordinal);
-</code></pre>
-<pre><code>/**
- * Get array child element data type name by ordinal
- *
- * @param ordinal the data index of carbon schema
- * @return ordinal array child element data type name
- */
-char *getArrayElementTypeName(int ordinal);
-</code></pre>
-<h3>
-<a id="carbonproperties" class="anchor" href="#carbonproperties" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonProperties</h3>
-<pre><code>/**
- * Constructor of CarbonProperties
- *
- * @param env JNI env
- */
-CarbonProperties(JNIEnv *env);
-</code></pre>
-<pre><code>/**
- * This method will be used to add a new property
- *
- * @param key property key
- * @param value property value
- * @return CarbonProperties object
- */
-jobject addProperty(char *key, char *value);
-</code></pre>
-<pre><code>/**
- * This method will be used to get the properties value
- *
- * @param key property key
- * @return property value
- */
-char *getProperty(char *key);
-</code></pre>
-<pre><code>/**
- * This method will be used to get the properties value
- * if property is not present then it will return the default value
- *
- * @param key property key
- * @param defaultValue property default Value
- * @return
- */
-char *getProperty(char *key, char *defaultValue);
-</code></pre>
-<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__api').addClass('selected');
-
- if (!$('.b-nav__api').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display api subnav items
- $('.b-nav__api').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
-</script></div>
-</div>
-</div>
-</div>
-<div class="doc-footer">
- <a href="#top" class="scroll-top">Top</a>
-</div>
-</div>
-</section>
-</div>
-</div>
-</div>
-</section><!-- End systemblock part -->
-<script src="js/custom.js"></script>
-</body>
-</html>
\ No newline at end of file
diff --git a/content/WEB-INF/classes/META-INF/NOTICE b/content/WEB-INF/classes/META-INF/NOTICE
index 0d64141..4a4f784 100644
--- a/content/WEB-INF/classes/META-INF/NOTICE
+++ b/content/WEB-INF/classes/META-INF/NOTICE
@@ -1,6 +1,6 @@
Apache CarbonData :: Website
-Copyright 2019 The Apache Software Foundation
+Copyright 2020 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
diff --git a/content/WEB-INF/classes/application.conf b/content/WEB-INF/classes/application.conf
index eb1c327..970edf4 100644
--- a/content/WEB-INF/classes/application.conf
+++ b/content/WEB-INF/classes/application.conf
@@ -6,7 +6,7 @@ fileList=["configuration-parameters",
"supported-data-types-in-carbondata",
"performance-tuning",
"sdk-guide",
- "datamap-developer-guide",
+ "index-developer-guide",
"ddl-of-carbondata",
"dml-of-carbondata",
"documentation",
@@ -22,17 +22,23 @@ fileList=["configuration-parameters",
"alluxio-guide",
"hive-guide",
"index-server",
- "presto-guide"
+
+ "flink-integration-guide",
+ "mv-guide",
+ "prestodb-guide",
+ "prestosql-guide",
+ "scd-and-cdc-guide",
+ "spatial-index-guide",
+
+
]
dataMapFileList=[
- "bloomfilter-datamap-guide",
- "lucene-datamap-guide",
- "preaggregate-datamap-guide",
- "mv-datamap-guide",
- "timeseries-datamap-guide",
- "datamap-management"]
+ "bloomfilter-index-guide",
+ "lucene-index-guide",
+ "secondary-index-guide",
+ "index-management"]
-dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/datamap/"
+dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/index/"
fileListToRetain=["quick-start-guide"
]
diff --git a/content/alluxio-guide.html b/content/alluxio-guide.html
index ead4da4..a470301 100644
--- a/content/alluxio-guide.html
+++ b/content/alluxio-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -262,7 +271,7 @@
<h3>
<a id="running-example" class="anchor" href="#running-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running Example</h3>
<ul>
-<li>Please refer to <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/AlluxioExample.scala" target=_blank>AlluxioExample</a>
+<li>Please refer to <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/scala/org/apache/carbondata/examples/AlluxioExample.scala" target=_blank>AlluxioExample</a>
</li>
</ul>
<h2>
@@ -290,7 +299,7 @@
<ul>
<li>Running the command in spark path</li>
</ul>
-<pre lang="$command"><code>./bin/spark-shell --jars ${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar,${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar
+<pre lang="$command"><code>./bin/spark-shell --jars ${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-2.0.0-SNAPSHOT-bin-spark2.3.4-hadoop2.7.2.jar,${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar
</code></pre>
<ul>
<li>Testing use alluxio by CarbonSession</li>
@@ -300,7 +309,7 @@ import org.apache.spark.sql.SparkSession
val carbon = SparkSession.builder().master("local").appName("test").getOrCreateCarbonSession("alluxio://localhost:19998/carbondata");
carbon.sql("CREATE TABLE carbon_alluxio(id String,name String, city String,age Int) STORED as carbondata");
-carbon.sql(s"LOAD DATA LOCAL INPATH '${CARBONDATA_PATH}/integration/spark-common-test/src/test/resources/sample.csv' into table carbon_alluxio");
+carbon.sql(s"LOAD DATA LOCAL INPATH '${CARBONDATA_PATH}/integration/spark/src/test/resources/sample.csv' into table carbon_alluxio");
carbon.sql("select * from carbon_alluxio").show
</code></pre>
<ul>
@@ -348,9 +357,9 @@ carbon.sql("select * from carbon_alluxio").show
<a id="command" class="anchor" href="#command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Command</h4>
<pre lang="$command"><code>./bin/spark-submit \
--master local \
---jars ${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar,${CARBONDATA_PATH}/examples/spark2/target/carbondata-examples-1.6.0-SNAPSHOT.jar \
+--jars ${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar,${CARBONDATA_PATH}/examples/spark/target/carbondata-examples-2.0.0-SNAPSHOT.jar \
--class org.apache.carbondata.examples.AlluxioExample \
-${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar \
+${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-2.0.0-SNAPSHOT-bin-spark2.3.4-hadoop2.7.2.jar \
false
</code></pre>
<p><strong>NOTE</strong>: Please set runShell as false, which can avoid dependency on alluxio shell module.</p>
diff --git a/content/bloomfilter-datamap-guide.html b/content/bloomfilter-index-guide.html
similarity index 74%
rename from content/bloomfilter-datamap-guide.html
rename to content/bloomfilter-index-guide.html
index 2231c50..2ff3289 100644
--- a/content/bloomfilter-datamap-guide.html
+++ b/content/bloomfilter-index-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,50 +240,53 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-bloomfilter-datamap" class="anchor" href="#carbondata-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter DataMap</h1>
+<a id="carbondata-bloomfilter-index" class="anchor" href="#carbondata-bloomfilter-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter Index</h1>
<ul>
-<li><a href="#datamap-management">DataMap Management</a></li>
-<li><a href="#bloomfilter-datamap-introduction">BloomFilter Datamap Introduction</a></li>
+<li><a href="#index-management">Index Management</a></li>
+<li><a href="#bloomfilter-index-introduction">BloomFilter Index Introduction</a></li>
<li><a href="#loading-data">Loading Data</a></li>
<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#data-management-with-bloomfilter-datamap">Data Management</a></li>
+<li><a href="#data-management-with-bloomfilter-index">Data Management</a></li>
<li><a href="#useful-tips">Useful Tips</a></li>
</ul>
<h4>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
-<p>Creating BloomFilter DataMap</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-ON TABLE main_table
-USING 'bloomfilter'
-DMPROPERTIES ('index_columns'='city, name', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
+<a id="index-management" class="anchor" href="#index-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Management</h4>
+<p>Creating BloomFilter Index</p>
+<pre><code>CREATE INDEX [IF NOT EXISTS] index_name
+ON TABLE main_table (city,name)
+AS 'bloomfilter'
+PROPERTIES ('BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
</code></pre>
-<p>Dropping Specified DataMap</p>
-<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
-ON TABLE main_table
+<p>Dropping Specified Index</p>
+<pre><code>DROP INDEX [IF EXISTS] index_name
+ON [TABLE] main_table
</code></pre>
-<p>Showing all DataMaps on this table</p>
-<pre><code>SHOW DATAMAP
-ON TABLE main_table
+<p>Showing all Indexes on this table</p>
+<pre><code>SHOW INDEXES
+ON [TABLE] main_table
</code></pre>
-<p>Disable DataMap</p>
<blockquote>
-<p>The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
+<p>NOTE: Keywords given inside <code>[]</code> is optional.</p>
+</blockquote>
+<p>Disable Index</p>
+<blockquote>
+<p>The index by default is enabled. To support tuning on query, we can disable a specific index during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
</blockquote>
-<pre><code>// disable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = false
-// enable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = true
+<pre><code>// disable the index
+SET carbon.index.visible.dbName.tableName.indexName = false
+// enable the index
+SET carbon.index.visible.dbName.tableName.indexName = true
</code></pre>
<h2>
-<a id="bloomfilter-datamap-introduction" class="anchor" href="#bloomfilter-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter DataMap Introduction</h2>
+<a id="bloomfilter-index-introduction" class="anchor" href="#bloomfilter-index-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter Index Introduction</h2>
<p>A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.
-Carbondata introduced BloomFilter as an index datamap to enhance the performance of querying with precise value.
-It is well suitable for queries that do precise match on high cardinality columns(such as Name/ID).
+Carbondata introduced BloomFilter as an index to enhance the performance of querying with precise value.
+It is well suitable for queries that do precise matching on high cardinality columns(such as Name/ID).
Internally, CarbonData maintains a BloomFilter per blocklet for each index column to indicate that whether a value of the column is in this blocklet.
-Just like the other datamaps, BloomFilter datamap is managed along with main tables by CarbonData.
-User can create BloomFilter datamap on specified columns with specified BloomFilter configurations such as size and probability.</p>
-<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
-<pre><code>CREATE TABLE datamap_test (
+Just like the other indexes, BloomFilter index is managed along with main tables by CarbonData.
+User can create BloomFilter index on specified columns with specified BloomFilter configurations such as size and probability.</p>
+<p>For instance, main table called <strong>index_test</strong> which is defined as:</p>
+<pre><code>CREATE TABLE index_test (
id string,
name string,
age int,
@@ -285,20 +297,21 @@ TBLPROPERTIES('SORT_COLUMNS'='id')
</code></pre>
<p>In the above example, <code>id</code> and <code>name</code> are high cardinality columns
and we always query on <code>id</code> and <code>name</code> with precise value.
-since <code>id</code> is in the sort_columns and it is orderd,
+since <code>id</code> is in the sort_columns and it is ordered,
query on it will be fast because CarbonData can skip all the irrelative blocklets.
But queries on <code>name</code> may be bad since the blocklet minmax may not help,
because in each blocklet the range of the value of <code>name</code> may be the same -- all from A* to z*.
-In this case, user can create a BloomFilter DataMap on column <code>name</code>.
-Moreover, user can also create a BloomFilter DataMap on the sort_columns.
+In this case, user can create a BloomFilter Index on column <code>name</code>.
+Moreover, user can also create a BloomFilter Index on the sort_columns.
This is useful if user has too many segments and the range of the value of sort_columns are almost the same.</p>
-<p>User can create BloomFilter DataMap using the Create DataMap DDL:</p>
-<pre><code>CREATE DATAMAP dm
-ON TABLE datamap_test
-USING 'bloomfilter'
-DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001', 'BLOOM_COMPRESS'='true')
+<p>User can create BloomFilter Index using the Create Index DDL:</p>
+<pre><code>CREATE INDEX dm
+ON TABLE index_test (name,id)
+AS 'bloomfilter'
+PROPERTIES ('BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001', 'BLOOM_COMPRESS'='true')
</code></pre>
-<p><strong>Properties for BloomFilter DataMap</strong></p>
+<p>Here, (name,id) are INDEX_COLUMNS. Carbondata will generate BloomFilter index on these columns. Queries on these columns are usually like <code>'COL = VAL'</code>.</p>
+<p><strong>Properties for BloomFilter Index</strong></p>
<table>
<thead>
<tr>
@@ -310,12 +323,6 @@ DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0
</thead>
<tbody>
<tr>
-<td>INDEX_COLUMNS</td>
-<td>YES</td>
-<td></td>
-<td>Carbondata will generate BloomFilter index on these columns. Queries on these columns are usually like 'COL = VAL'.</td>
-</tr>
-<tr>
<td>BLOOM_SIZE</td>
<td>NO</td>
<td>640000</td>
@@ -338,43 +345,43 @@ DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0
<h2>
<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data</h2>
<p>When loading data to main table, BloomFilter files will be generated for all the
-index_columns given in DMProperties which contains the blockletId and a BloomFilter for each index column.
-These index files will be written inside a folder named with DataMap name
+index_columns provided in the CREATE statement which contains the blockletId and a BloomFilter for each index column.
+These index files will be written inside a folder named with Index name
inside each segment folders.</p>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying Data</h2>
-<p>User can verify whether a query can leverage BloomFilter DataMap by executing <code>EXPLAIN</code> command,
-which will show the transformed logical plan, and thus user can check whether the BloomFilter DataMap can skip blocklets during the scan.
-If the DataMap does not prune blocklets well, you can try to increase the value of property <code>BLOOM_SIZE</code> and decrease the value of property <code>BLOOM_FPP</code>.</p>
+<p>User can verify whether a query can leverage BloomFilter Index by executing <code>EXPLAIN</code> command,
+which will show the transformed logical plan, and thus user can check whether the BloomFilter Index can skip blocklets during the scan.
+If the Index does not prune blocklets well, you can try to increase the value of property <code>BLOOM_SIZE</code> and decrease the value of property <code>BLOOM_FPP</code>.</p>
<h2>
-<a id="data-management-with-bloomfilter-datamap" class="anchor" href="#data-management-with-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management With BloomFilter DataMap</h2>
-<p>Data management with BloomFilter DataMap has no difference with that on Lucene DataMap.
-You can refer to the corresponding section in <code>CarbonData Lucene DataMap</code>.</p>
+<a id="data-management-with-bloomfilter-index" class="anchor" href="#data-management-with-bloomfilter-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management With BloomFilter Index</h2>
+<p>Data management with BloomFilter Index has no difference with that on Lucene Index.
+You can refer to the corresponding section in <a href="https://github.com/apache/carbondata/blob/master/docs/index/lucene-index-guide.html" target=_blank>CarbonData Lucene Index</a></p>
<h2>
<a id="useful-tips" class="anchor" href="#useful-tips" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Useful Tips</h2>
<ul>
-<li>BloomFilter DataMap is suggested to be created on the high cardinality columns.
+<li>BloomFilter Index is suggested to be created on the high cardinality columns.
Query conditions on these columns are always simple <code>equal</code> or <code>in</code>,
such as 'col1=XX', 'col1 in (XX, YY)'.</li>
-<li>We can create multiple BloomFilter DataMaps on one table,
-but we do recommend you to create one BloomFilter DataMap that contains multiple index columns,
+<li>We can create multiple BloomFilter Indexes on one table,
+but we do recommend you to create one BloomFilter Index that contains multiple index columns,
because the data loading and query performance will be better.</li>
<li>
-<code>BLOOM_FPP</code> is only the expected number from user, the actually FPP may be worse.
-If the BloomFilter DataMap does not work well,
+<code>BLOOM_FPP</code> is only the expected number from user, the actual FPP may be worse.
+If the BloomFilter Index does not work well,
you can try to increase <code>BLOOM_SIZE</code> and decrease <code>BLOOM_FPP</code> at the same time.
Notice that bigger <code>BLOOM_SIZE</code> will increase the size of index file
and smaller <code>BLOOM_FPP</code> will increase runtime calculation while performing query.</li>
-<li>'0' skipped blocklets of BloomFilter DataMap in explain output indicates that
-BloomFilter DataMap does not prune better than Main DataMap.
-(For example since the data is not ordered, a specific value may be contained in many blocklets. In this case, bloom may not work better than Main DataMap.)
+<li>'0' skipped blocklets of BloomFilter Index in explain output indicates that
+BloomFilter Index does not prune better than Main Index.
+(For example since the data is not ordered, a specific value may be contained in many blocklets. In this case, bloom may not work better than Main Index.)
If this occurs very often, it means that current BloomFilter is useless. You can disable or drop it.
-Sometimes we cannot see any pruning result about BloomFilter DataMap in the explain output,
-this indicates that the previous DataMap has pruned all the blocklets and there is no need to continue pruning.</li>
-<li>In some scenarios, the BloomFilter DataMap may not enhance the query performance significantly
+Sometimes we cannot see any pruning result about BloomFilter Index in the explain output,
+this indicates that the previous Index has pruned all the blocklets and there is no need to continue pruning.</li>
+<li>In some scenarios, the BloomFilter Index may not enhance the query performance significantly
but if it can reduce the number of spark task,
-there is still a chance that BloomFilter DataMap can enhance the performance for concurrent query.</li>
-<li>Note that BloomFilter DataMap will decrease the data loading performance and may cause slightly storage expansion (for DataMap index file).</li>
+there is still a chance that BloomFilter Index can enhance the performance for concurrent query.</li>
+<li>Note that BloomFilter Index will decrease the data loading performance and may cause slight storage expansion (for index file).</li>
</ul>
<script>
$(function() {
diff --git a/content/carbon-as-spark-datasource-guide.html b/content/carbon-as-spark-datasource-guide.html
index 4e03b69..de6aabd 100644
--- a/content/carbon-as-spark-datasource-guide.html
+++ b/content/carbon-as-spark-datasource-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -304,7 +313,7 @@
<p><strong>NOTE:</strong> please set long_string_columns for varchar column.</p>
<h2>
<a id="example" class="anchor" href="#example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example</h2>
-<pre><code> CREATE TABLE CARBON_TABLE (NAME STRING) USING CARBON OPTIONS('table_block_size'='256')
+<pre><code> CREATE TABLE CARBON_TABLE (NAME STRING) USING CARBON OPTIONS('table_blocksize'='256')
</code></pre>
<h1>
<a id="using-dataframe" class="anchor" href="#using-dataframe" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using DataFrame</h1>
diff --git a/content/configuration-parameters.html b/content/configuration-parameters.html
index 2519009..9293342 100644
--- a/content/configuration-parameters.html
+++ b/content/configuration-parameters.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -254,11 +263,6 @@
</thead>
<tbody>
<tr>
-<td>carbon.storelocation</td>
-<td>spark.sql.warehouse.dir property value</td>
-<td>Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. <strong>NOTE:</strong> Store location should be in HDFS or S3.</td>
-</tr>
-<tr>
<td>carbon.ddl.base.hdfs.url</td>
<td>(none)</td>
<td>To simplify and shorten the path to be specified in DDL/DML commands, this property is supported. This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS of core-site.xml. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://1 [...]
@@ -271,7 +275,7 @@
<tr>
<td>carbon.streaming.auto.handoff.enabled</td>
<td>true</td>
-<td>CarbonData supports storing of streaming data. To have high throughput for streaming, the data is written in Row format which is highly optimized for write, but performs poorly for query. When this property is true and when the streaming data size reaches <em><strong>carbon.streaming.segment.max.size</strong></em>, CabonData will automatically convert the data to columnar format and optimize it for faster querying.<strong>NOTE:</strong> It is not recommended to keep the default value [...]
+<td>CarbonData supports storing of streaming data. To have high throughput for streaming, the data is written in Row format which is highly optimized for write, but performs poorly for query. When this property is true and when the streaming data size reaches <em><strong>carbon.streaming.segment.max.size</strong></em>, CabonData will automatically convert the data to columnar format and optimize it for faster querying. <strong>NOTE:</strong> It is not recommended to keep the default valu [...]
</tr>
<tr>
<td>carbon.streaming.segment.max.size</td>
@@ -279,11 +283,6 @@
<td>CarbonData writes streaming data in row format which is optimized for high write throughput. This property defines the maximum size of data to be held is row format, beyond which it will be converted to columnar format in order to support high performance query, provided <em><strong>carbon.streaming.auto.handoff.enabled</strong></em> is true. <strong>NOTE:</strong> Setting higher value will impact the streaming ingestion. The value has to be configured in bytes.</td>
</tr>
<tr>
-<td>carbon.query.show.datamaps</td>
-<td>true</td>
-<td>CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.<strong>NOTE:</strong> It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen. But it is shown by default so that user or ad [...]
-</tr>
-<tr>
<td>carbon.segment.lock.files.preserve.hours</td>
<td>48</td>
<td>In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments. Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours.</td>
@@ -316,7 +315,7 @@
<tr>
<td>carbon.unsafe.driver.working.memory.in.mb</td>
<td>(none)</td>
-<td>CarbonData supports storing data in unsafe on-heap memory in driver for certain operations like insert into, query for loading datamap cache. The Minimum value recommended is 512MB. If this configuration is not set, carbondata will use the value of <code>carbon.unsafe.working.memory.in.mb</code>.</td>
+<td>CarbonData supports storing data in unsafe on-heap memory in driver for certain operations like insert into, query for loading index cache. The Minimum value recommended is 512MB. If this configuration is not set, carbondata will use the value of <code>carbon.unsafe.working.memory.in.mb</code>.</td>
</tr>
<tr>
<td>carbon.update.sync.folder</td>
@@ -343,6 +342,16 @@
<td>None</td>
<td>To support FileTypeInterface for configuring custom CarbonFile implementation to work with custom FileSystem.</td>
</tr>
+<tr>
+<td>carbon.timeseries.first.day.of.week</td>
+<td>SUNDAY</td>
+<td>This parameter configures which day of the week to be considered as first day of the week. Because first day of the week will be different in different parts of the world.</td>
+</tr>
+<tr>
+<td>carbon.enable.tablestatus.backup</td>
+<td>false</td>
+<td>In cloud object store scenario, overwriting table status file is not an atomic operation since it uses rename API. Thus, it is possible that table status is corrupted if process crashed when overwriting the table status file. To protect from file corruption, user can enable this property.</td>
+</tr>
</tbody>
</table>
<h2>
@@ -374,17 +383,17 @@
<tr>
<td>carbon.loading.prefetch</td>
<td>false</td>
-<td>CarbonData uses univocity parser to read csv files. This configuration is used to inform the parser whether it can prefetch the data from csv files to speed up the reading.<strong>NOTE:</strong> Enabling prefetch improves the data loading performance, but needs higher memory to keep more records which are read ahead from disk.</td>
+<td>CarbonData uses univocity parser to read csv files. This configuration is used to inform the parser whether it can prefetch the data from csv files to speed up the reading. <strong>NOTE:</strong> Enabling prefetch improves the data loading performance, but needs higher memory to keep more records which are read ahead from disk.</td>
</tr>
<tr>
<td>carbon.skip.empty.line</td>
<td>false</td>
-<td>The csv files givent to CarbonData for loading can contain empty lines. Based on the business scenario, this empty line might have to be ignored or needs to be treated as NULL value for all columns. In order to define this business behavior, this configuration is provided.<strong>NOTE:</strong> In order to consider NULL values for non string columns and continue with data load, <em><strong>carbon.bad.records.action</strong></em> need to be set to <strong>FORCE</strong>;else data load [...]
+<td>The csv files givent to CarbonData for loading can contain empty lines. Based on the business scenario, this empty line might have to be ignored or needs to be treated as NULL value for all columns. In order to define this business behavior, this configuration is provided. <strong>NOTE:</strong> In order to consider NULL values for non string columns and continue with data load, <em><strong>carbon.bad.records.action</strong></em> need to be set to <strong>FORCE</strong>;else data loa [...]
</tr>
<tr>
<td>carbon.number.of.cores.while.loading</td>
<td>2</td>
-<td>Number of cores to be used while loading data. This also determines the number of threads to be used to read the input files (csv) in parallel.<strong>NOTE:</strong> This configured value is used in every data loading step to parallelize the operations. Configuring a higher value can lead to increased early thread pre-emption by OS and there by reduce the overall performance.</td>
+<td>Number of cores to be used while loading data. This also determines the number of threads to be used to read the input files (csv) in parallel. <strong>NOTE:</strong> This configured value is used in every data loading step to parallelize the operations. Configuring a higher value can lead to increased early thread pre-emption by OS and there by reduce the overall performance.</td>
</tr>
<tr>
<td>enable.unsafe.sort</td>
@@ -398,13 +407,8 @@
</tr>
<tr>
<td>carbon.load.sort.scope</td>
-<td>LOCAL_SORT</td>
-<td>CarbonData can support various sorting options to match the balance between load and query performance. LOCAL_SORT:All the data given to an executor in the single load is fully sorted and written to carbondata files. Data loading performance is reduced a little as the entire data needs to be sorted in the executor. BATCH_SORT:Sorts the data in batches of configured size and writes to carbondata files. Data loading performance increases as the entire data need not be sorted. But query [...]
-</tr>
-<tr>
-<td>carbon.load.batch.sort.size.inmb</td>
-<td>0</td>
-<td>When <em><strong>carbon.load.sort.scope</strong></em> is configured as <em><strong>BATCH_SORT</strong></em>, this configuration needs to be added to specify the batch size for sorting and writing to carbondata files. <strong>NOTE:</strong> It is recommended to keep the value around 45% of <em><strong>carbon.sort.storage.inmemory.size.inmb</strong></em> to avoid spill to disk. Also it is recommended to keep the value higher than <em><strong>carbon.blockletgroup.size.in.mb</strong></e [...]
+<td>NO_SORT [If sort columns are not specified while creating table] and LOCAL_SORT [If sort columns are specified]</td>
+<td>CarbonData can support various sorting options to match the balance between load and query performance. LOCAL_SORT: All the data given to an executor in the single load is fully sorted and written to carbondata files. Data loading performance is reduced a little as the entire data needs to be sorted in the executor. GLOBAL SORT: Entire data in the data load is fully sorted and written to carbondata files. Data loading performance would get reduced as the entire data needs to be sorte [...]
</tr>
<tr>
<td>carbon.global.sort.rdd.storage.level</td>
@@ -472,11 +476,6 @@
<td>When the configuration <em><strong>carbon.merge.sort.prefetch</strong></em> is configured to true, we need to set the number of records that can be prefetched. This configuration is used specify the number of records to be prefetched.**NOTE: **Configuring more number of records to be prefetched increases memory footprint as more records will have to be kept in memory.</td>
</tr>
<tr>
-<td>enable.inmemory.merge.sort</td>
-<td>false</td>
-<td>CarbonData sorts and writes data to intermediate files to limit the memory usage. These intermediate files needs to be sorted again using merge sort before writing to the final carbondata file. Performing merge sort in memory would increase the sorting performance at the cost of increased memory footprint. This Configuration specifies to do in-memory merge sort or to do file based merge sort.</td>
-</tr>
-<tr>
<td>carbon.sort.storage.inmemory.size.inmb</td>
<td>512</td>
<td>CarbonData writes every <em><strong>carbon.sort.size</strong></em> number of records to intermediate temp files during data loading to ensure memory footprint is within limits. When <em><strong>enable.unsafe.sort</strong></em> configuration is enabled, instead of using <em><strong>carbon.sort.size</strong></em> which is based on rows count, size occupied in memory is used to determine when to flush data pages to intermediate temp files. This configuration determines the memory to be [...]
@@ -515,7 +514,7 @@
<tr>
<td>carbon.load.skewedDataOptimization.enabled</td>
<td>false</td>
-<td>During data loading,CarbonData would divide the number of blocks equally so as to ensure all executors process same number of blocks. This mechanism satisfies most of the scenarios and ensures maximum parallel processing for optimal data loading performance. In some business scenarios, there might be scenarios where the size of blocks vary significantly and hence some executors would have to do more work if they get blocks containing more data. This configuration enables size based b [...]
+<td>During data loading,CarbonData would divide the number of blocks equally so as to ensure all executors process same number of blocks. This mechanism satisfies most of the scenarios and ensures maximum parallel processing for optimal data loading performance. In some business scenarios, there might be scenarios where the size of blocks vary significantly and hence some executors would have to do more work if they get blocks containing more data. This configuration enables size based b [...]
</tr>
<tr>
<td>enable.data.loading.statistics</td>
@@ -528,19 +527,9 @@
<td>CarbonData generates dictionary keys and writes them to separate dictionary file during data loading. To optimize the IO, this configuration determines the number of dictionary keys to be persisted to dictionary file at a time. <strong>NOTE:</strong> Writing to file also serves as a commit point to the dictionary generated. Increasing more values in memory causes more data loss during system or application failure. It is advised to alter this configuration judiciously.</td>
</tr>
<tr>
-<td>dictionary.worker.threads</td>
-<td>1</td>
-<td>CarbonData supports Optimized data loading by relying on a dictionary server. Dictionary server helps to maintain dictionary values independent of the data loading and there by avoids reading the same input data multiples times. This configuration determines the number of concurrent dictionary generation or request that needs to be served by the dictionary server. <strong>NOTE:</strong> This configuration takes effect when <em><strong>carbon.options.single.pass</strong></em> is confi [...]
-</tr>
-<tr>
-<td>carbon.dictionary.server.port</td>
-<td>2030</td>
-<td>Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. Single pass loading can be enabled using the option <em><strong>carbon.options.single.pass</strong></em>. When this option is specified, a dictionary server will be internally started to handle the dictionary generation and query requests. [...]
-</tr>
-<tr>
<td>carbon.load.directWriteToStorePath.enabled</td>
<td>false</td>
-<td>During data load, all the carbondata files are written to local disk and finally copied to the target store location in HDFS/S3. Enabling this parameter will make carbondata files to be written directly onto target HDFS/S3 location bypassing the local disk.<strong>NOTE:</strong> Writing directly to HDFS/S3 saves local disk IO(once for writing the files and again for copying to HDFS/S3) there by improving the performance. But the drawback is when data loading fails or the application [...]
+<td>During data load, all the carbondata files are written to local disk and finally copied to the target store location in HDFS/S3. Enabling this parameter will make carbondata files to be written directly onto target HDFS/S3 location bypassing the local disk. <strong>NOTE:</strong> Writing directly to HDFS/S3 saves local disk IO(once for writing the files and again for copying to HDFS/S3) there by improving the performance. But the drawback is when data loading fails or the application [...]
</tr>
<tr>
<td>carbon.options.serialization.null.format</td>
@@ -572,6 +561,11 @@
<td>4</td>
<td>size based threshold for local dictionary in MB, maximum allowed size is 16 MB.</td>
</tr>
+<tr>
+<td>carbon.enable.bad.record.handling.for.insert</td>
+<td>false</td>
+<td>by default, disable the bad record and converter step during "insert into"</td>
+</tr>
</tbody>
</table>
<h2>
@@ -593,7 +587,7 @@
<tr>
<td>carbon.compaction.level.threshold</td>
<td>4, 3</td>
-<td>Each CarbonData load will create one segment, if every load is small in size it will generate many small file over a period of time impacting the query performance. This configuration is for minor compaction which decides how many segments to be merged. Configuration is of the form (x,y). Compaction will be triggered for every x segments and form a single level 1 compacted segment. When the number of compacted level 1 segments reach y, compaction will be triggered again to merge them [...]
+<td>Each CarbonData load will create one segment, if every load is small in size it will generate many small file over a period of time impacting the query performance. This configuration is for minor compaction which decides how many segments to be merged. Configuration is of the form (x,y). Compaction will be triggered for every x segments and form a single level 1 compacted segment. When the number of compacted level 1 segments reach y, compaction will be triggered again to merge them [...]
</tr>
<tr>
<td>carbon.major.compaction.size</td>
@@ -603,7 +597,7 @@
<tr>
<td>carbon.horizontal.compaction.enable</td>
<td>true</td>
-<td>CarbonData supports DELETE/UPDATE functionality by creating delta data files for existing carbondata files. These delta files would grow as more number of DELETE/UPDATE operations are performed. Compaction of these delta files are termed as horizontal compaction. This configuration is used to turn ON/OFF horizontal compaction. After every DELETE and UPDATE statement, horizontal compaction may occur in case the delta (DELETE/ UPDATE) files becomes more than specified threshold.**NOTE: [...]
+<td>CarbonData supports DELETE/UPDATE functionality by creating delta data files for existing carbondata files. These delta files would grow as more number of DELETE/UPDATE operations are performed. Compaction of these delta files are termed as horizontal compaction. This configuration is used to turn ON/OFF horizontal compaction. After every DELETE and UPDATE statement, horizontal compaction may occur in case the delta (DELETE/ UPDATE) files becomes more than specified threshold. <stron [...]
</td>
</tr>
<tr>
@@ -619,48 +613,53 @@
<tr>
<td>carbon.update.segment.parallelism</td>
<td>1</td>
-<td>CarbonData processes the UPDATE operations by grouping records belonging to a segment into a single executor task. When the amount of data to be updated is more, this behavior causes problems like restarting of executor due to low memory and data-spill related errors. This property specifies the parallelism for each segment during update.<strong>NOTE:</strong> It is recommended to set this value to a multiple of the number of executors for balance. Values range between 1 to 1000.</td>
+<td>CarbonData processes the UPDATE operations by grouping records belonging to a segment into a single executor task. When the amount of data to be updated is more, this behavior causes problems like restarting of executor due to low memory and data-spill related errors. This property specifies the parallelism for each segment during update. <strong>NOTE:</strong> It is recommended to set this value to a multiple of the number of executors for balance. Values range between 1 to 1000.</td>
</tr>
<tr>
<td>carbon.numberof.preserve.segments</td>
<td>0</td>
-<td>If the user wants to preserve some number of segments from being compacted then he can set this configuration. Example: carbon.numberof.preserve.segments = 2 then 2 latest segments will always be excluded from the compaction. No segments will be preserved by default.<strong>NOTE:</strong> This configuration is useful when the chances of input data can be wrong due to environment scenarios. Preserving some of the latest segments from being compacted can help to easily delete the wrong [...]
+<td>If the user wants to preserve some number of segments from being compacted then he can set this configuration. Example: carbon.numberof.preserve.segments = 2 then 2 latest segments will always be excluded from the compaction. No segments will be preserved by default. <strong>NOTE:</strong> This configuration is useful when the chances of input data can be wrong due to environment scenarios. Preserving some of the latest segments from being compacted can help to easily delete the wron [...]
</tr>
<tr>
<td>carbon.allowed.compaction.days</td>
<td>0</td>
-<td>This configuration is used to control on the number of recent segments that needs to be compacted, ignoring the older ones. This configuration is in days. For Example: If the configuration is 2, then the segments which are loaded in the time frame of past 2 days only will get merged. Segments which are loaded earlier than 2 days will not be merged. This configuration is disabled by default.<strong>NOTE:</strong> This configuration is useful when a bulk of history data is loaded into [...]
+<td>This configuration is used to control on the number of recent segments that needs to be compacted, ignoring the older ones. This configuration is in days. For Example: If the configuration is 2, then the segments which are loaded in the time frame of past 2 days only will get merged. Segments which are loaded earlier than 2 days will not be merged. This configuration is disabled by default. <strong>NOTE:</strong> This configuration is useful when a bulk of history data is loaded into [...]
</tr>
<tr>
<td>carbon.enable.auto.load.merge</td>
<td>false</td>
-<td>Compaction can be automatically triggered once data load completes. This ensures that the segments are merged in time and thus query times does not increase with increase in segments. This configuration enables to do compaction along with data loading.**NOTE: **Compaction will be triggered once the data load completes. But the status of data load wait till the compaction is completed. Hence it might look like data loading time has increased, but thats not the case. Moreover failure o [...]
+<td>Compaction can be automatically triggered once data load completes. This ensures that the segments are merged in time and thus query times does not increase with increase in segments. This configuration enables to do compaction along with data loading. <strong>NOTE:</strong> Compaction will be triggered once the data load completes. But the status of data load wait till the compaction is completed. Hence it might look like data loading time has increased, but thats not the case. More [...]
</tr>
<tr>
<td>carbon.enable.page.level.reader.in.compaction</td>
-<td>true</td>
+<td>false</td>
<td>Enabling page level reader for compaction reduces the memory usage while compacting more number of segments. It allows reading only page by page instead of reading whole blocklet to memory. <strong>NOTE:</strong> Please refer to <a href="./file-structure-of-carbondata.html#carbondata-file-format">file-structure-of-carbondata</a> to understand the storage format of CarbonData and concepts of pages.</td>
</tr>
<tr>
<td>carbon.concurrent.compaction</td>
<td>true</td>
-<td>Compaction of different tables can be executed concurrently. This configuration determines whether to compact all qualifying tables in parallel or not. **NOTE: **Compacting concurrently is a resource demanding operation and needs more resources there by affecting the query performance also. This configuration is <strong>deprecated</strong> and might be removed in future releases.</td>
+<td>Compaction of different tables can be executed concurrently. This configuration determines whether to compact all qualifying tables in parallel or not. <strong>NOTE:</strong> Compacting concurrently is a resource demanding operation and needs more resources there by affecting the query performance also. This configuration is <strong>deprecated</strong> and might be removed in future releases.</td>
</tr>
<tr>
<td>carbon.compaction.prefetch.enable</td>
<td>false</td>
-<td>Compaction operation is similar to Query + data load where in data from qualifying segments are queried and data loading performed to generate a new single segment. This configuration determines whether to query ahead data from segments and feed it for data loading. **NOTE: **This configuration is disabled by default as it needs extra resources for querying extra data. Based on the memory availability on the cluster, user can enable it to improve compaction performance.</td>
+<td>Compaction operation is similar to Query + data load where in data from qualifying segments are queried and data loading performed to generate a new single segment. This configuration determines whether to query ahead data from segments and feed it for data loading. <strong>NOTE:</strong> This configuration is disabled by default as it needs extra resources for querying extra data. Based on the memory availability on the cluster, user can enable it to improve compaction performance.</td>
</tr>
<tr>
<td>carbon.merge.index.in.segment</td>
<td>true</td>
-<td>Each CarbonData file has a companion CarbonIndex file which maintains the metadata about the data. These CarbonIndex files are read and loaded into driver and is used subsequently for pruning of data during queries. These CarbonIndex files are very small in size(few KB) and are many. Reading many small files from HDFS is not efficient and leads to slow IO performance. Hence these CarbonIndex files belonging to a segment can be combined into a single file and read once there by incre [...]
+<td>Each CarbonData file has a companion CarbonIndex file which maintains the metadata about the data. These CarbonIndex files are read and loaded into driver and is used subsequently for pruning of data during queries. These CarbonIndex files are very small in size(few KB) and are many. Reading many small files from HDFS is not efficient and leads to slow IO performance. Hence these CarbonIndex files belonging to a segment can be combined into a single file and read once there by incre [...]
</tr>
<tr>
<td>carbon.enable.range.compaction</td>
<td>true</td>
<td>To configure Ranges-based Compaction to be used or not for RANGE_COLUMN. If true after compaction also the data would be present in ranges.</td>
</tr>
+<tr>
+<td>carbon.si.segment.merge</td>
+<td>false</td>
+<td>Making this true degrade the LOAD performance. When the number of small files increase for SI segments(it can happen as number of columns will be less and we store position id and reference columns), user an either set to true which will merge the data files for upcoming loads or run SI refresh command which does this job for all segments. (REFRESH INDEX <index_table>)</td>
+</tr>
</tbody>
</table>
<h2>
@@ -697,7 +696,7 @@
<tr>
<td>carbon.dynamical.location.scheduler.timeout</td>
<td>5</td>
-<td>CarbonData has its own scheduling algorithm to suggest to Spark on how many tasks needs to be launched and how much work each task need to do in a Spark cluster for any query on CarbonData. To determine the number of tasks that can be scheduled, knowing the count of active executors is necessary. When dynamic allocation is enabled on a YARN based spark cluster, executor processes are shutdown if no request is received for a particular amount of time. The executors are brought up when [...]
+<td>CarbonData has its own scheduling algorithm to suggest to Spark on how many tasks needs to be launched and how much work each task need to do in a Spark cluster for any query on CarbonData. To determine the number of tasks that can be scheduled, knowing the count of active executors is necessary. When dynamic allocation is enabled on a YARN based spark cluster, executor processes are shutdown if no request is received for a particular amount of time. The executors are brought up when [...]
</tr>
<tr>
<td>carbon.scheduler.min.registered.resources.ratio</td>
@@ -705,37 +704,6 @@
<td>Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0.</td>
</tr>
<tr>
-<td>carbon.search.enabled (Alpha Feature)</td>
-<td>false</td>
-<td>If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead.</td>
-</tr>
-<tr>
-<td>carbon.search.query.timeout</td>
-<td>10s</td>
-<td>Time within which the result is expected from the workers, beyond which the query is terminated</td>
-</tr>
-<tr>
-<td>carbon.search.scan.thread</td>
-<td>num of cores available in worker node</td>
-<td>Number of cores to be used in each worker for performing scan.</td>
-</tr>
-<tr>
-<td>carbon.search.master.port</td>
-<td>10020</td>
-<td>Port on which the search master listens for incoming query requests</td>
-</tr>
-<tr>
-<td>carbon.search.worker.port</td>
-<td>10021</td>
-<td>Port on which search master communicates with the workers.</td>
-</tr>
-<tr>
-<td>carbon.search.worker.workload.limit</td>
-<td>10 * <em>carbon.search.scan.thread</em>
-</td>
-<td>Maximum number of active requests that can be sent to a worker. Beyond which the request needs to be rescheduled for later time or to a different worker.</td>
-</tr>
-<tr>
<td>carbon.detail.batch.size</td>
<td>100</td>
<td>The buffer size to store records, returned from the block scan. In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000.<br><br> <strong>NOTE</stro [...]
@@ -758,7 +726,7 @@
<tr>
<td>enable.query.statistics</td>
<td>false</td>
-<td>CarbonData has extensive logging which would be useful for debugging issues related to performance or hard to locate issues. This configuration when made <em><strong>true</strong></em> would log additional query statistics information to more accurately locate the issues being debugged.<strong>NOTE:</strong> Enabling this would log more debug information to log files, there by increasing the log files size significantly in short span of time. It is advised to configure the log files [...]
+<td>CarbonData has extensive logging which would be useful for debugging issues related to performance or hard to locate issues. This configuration when made <em><strong>true</strong></em> would log additional query statistics information to more accurately locate the issues being debugged. <strong>NOTE:</strong> Enabling this would log more debug information to log files, there by increasing the log files size significantly in short span of time. It is advised to configure the log files [...]
</tr>
<tr>
<td>enable.unsafe.in.query.processing</td>
@@ -785,6 +753,31 @@
<td>true</td>
<td>By default this property is true, so prefetch is used in query to read next blocklet asynchronously in other thread while processing current blocklet in main thread. This can help to reduce CPU idle time. Setting this property false will disable this prefetch feature in query.</td>
</tr>
+<tr>
+<td>carbon.query.stage.input.enable</td>
+<td>false</td>
+<td>Stage input files are data files written by external applications (such as Flink), but have not been loaded into carbon table. Enabling this configuration makes query to include these files, thus makes query on latest data. However, since these files are not indexed, query maybe slower as full scan is required for these files.</td>
+</tr>
+<tr>
+<td>carbon.driver.pruning.multi.thread.enable.files.count</td>
+<td>100000</td>
+<td>To prune in multi-thread when total number of segment files for a query increases beyond the configured value.</td>
+</tr>
+<tr>
+<td>carbon.load.all.segment.indexes.to.cache</td>
+<td>true</td>
+<td>Setting this configuration to false, will prune and load only matched segment indexes to cache using segment metadata information such as columnid and it's minmax values, which decreases the usage of driver memory.</td>
+</tr>
+<tr>
+<td>carbon.secondary.index.creation.threads</td>
+<td>1</td>
+<td>Specifies the number of threads to concurrently process segments during secondary index creation. This property helps fine tuning the system when there are a lot of segments in a table. The value range is 1 to 50.</td>
+</tr>
+<tr>
+<td>carbon.si.lookup.partialstring</td>
+<td>true</td>
+<td>When true, it includes starts with, ends with and contains. When false, it includes only starts with secondary indexes.</td>
+</tr>
</tbody>
</table>
<h2>
@@ -799,16 +792,6 @@
</thead>
<tbody>
<tr>
-<td>carbon.insert.persist.enable</td>
-<td>false</td>
-<td>CarbonData does loading in 2 major steps.1st step reads from the input source and generates the dictionary values.2nd step reads from the source again and encodes the data with the dictionary values, perform index calculations and writes in CarbonData format. Suppose we are loading the CarbonData table using another table as source(using insert into) and the source table is being loaded in parallel, there can be cases where some data got inserted into the source table after CarbonDat [...]
-</tr>
-<tr>
-<td>carbon.insert.storage.level</td>
-<td>MEMORY_AND_DISK</td>
-<td>Storage level to persist dataset of a RDD/dataframe. Applicable when <em><strong>carbon.insert.persist.enable</strong></em> is <strong>true</strong>, if user's executor has less memory, set this parameter to 'MEMORY_AND_DISK_SER' or other storage level to correspond to different environment. <a href="http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence" rel="nofollow">See detail</a>.</td>
-</tr>
-<tr>
<td>carbon.update.persist.enable</td>
<td>true</td>
<td>Configuration to enable the dataset of RDD/dataframe to persist data. Enabling this will reduce the execution time of UPDATE operation.</td>
@@ -818,6 +801,11 @@
<td>MEMORY_AND_DISK</td>
<td>Storage level to persist dataset of a RDD/dataframe. Applicable when <em><strong>carbon.update.persist.enable</strong></em> is <strong>true</strong>, if user's executor has less memory, set this parameter to 'MEMORY_AND_DISK_SER' or other storage level to correspond to different environment. <a href="http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence" rel="nofollow">See detail</a>.</td>
</tr>
+<tr>
+<td>carbon.update.check.unique.value</td>
+<td>true</td>
+<td>By default this property is true, so update will validate key value mapping. This validation might have slight degrade in performance of update query. If user knows that key value mapping is correct, can disable this validation for better update performance by setting this property to false.</td>
+</tr>
</tbody>
</table>
<h2>
@@ -880,35 +868,23 @@
<tbody>
<tr>
<td>carbon.options.bad.records.logger.enable</td>
-<td>CarbonData can identify the records that are not conformant to schema and isolate them as bad records. Enabling this configuration will make CarbonData to log such bad records.<strong>NOTE:</strong> If the input data contains many bad records, logging them will slow down the over all data loading throughput. The data load operation status would depend on the configuration in <em><strong>carbon.bad.records.action</strong></em>.</td>
-</tr>
-<tr>
-<td>carbon.options.bad.records.logger.enable</td>
-<td>To enable or disable bad record logger.</td>
+<td>To enable or disable a bad record logger. CarbonData can identify the records that are not conformant to schema and isolate them as bad records. Enabling this configuration will make CarbonData to log such bad records. <strong>NOTE:</strong> If the input data contains many bad records, logging them will slow down the overall data loading throughput. The data load operation status would depend on the configuration in <em><strong>carbon.bad.records.action</strong></em>.</td>
</tr>
<tr>
<td>carbon.options.bad.records.action</td>
-<td>This property can have four types of actions for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found.</td>
+<td>This property has four types of bad record actions: FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found.</td>
</tr>
<tr>
<td>carbon.options.is.empty.data.bad.record</td>
<td>If false, then empty ("" or '' or ,,) data will not be considered as bad record and vice versa.</td>
</tr>
<tr>
-<td>carbon.options.batch.sort.size.inmb</td>
-<td>Size of batch data to keep in memory, as a thumb rule it supposed to be less than 45% of sort.inmemory.size.inmb otherwise it may spill intermediate data to disk.</td>
-</tr>
-<tr>
-<td>carbon.options.single.pass</td>
-<td>Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. This option specifies whether to use single pass for loading data or not. By default this option is set to FALSE. <strong>NOTE:</strong> Enabling this starts a new dictionary server to handle dictionary generation requests during data load [...]
-</tr>
-<tr>
<td>carbon.options.bad.record.path</td>
<td>Specifies the HDFS path where bad records needs to be stored.</td>
</tr>
<tr>
<td>carbon.custom.block.distribution</td>
-<td>Specifies whether to use the Spark or Carbon block distribution feature.**NOTE: **Refer to <a href="#query-configuration">Query Configuration</a>#carbon.custom.block.distribution for more details on CarbonData scheduler.</td>
+<td>Specifies whether to use the Spark or Carbon block distribution feature. <strong>NOTE:</strong> Refer to <a href="#query-configuration">Query Configuration</a>#carbon.custom.block.distribution for more details on CarbonData scheduler.</td>
</tr>
<tr>
<td>enable.unsafe.sort</td>
@@ -927,7 +903,7 @@
<td>Specifies how the current data load should be sorted with. This sort parameter is at the table level. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#carbon.sort.scope for detailed information.</td>
</tr>
<tr>
-<td>carbon.table.load.sort.scope.db_name.table_name</td>
+<td>carbon.table.load.sort.scope.<db_name>.<table_name></td>
<td>Overrides the SORT_SCOPE provided in CREATE TABLE.</td>
</tr>
<tr>
@@ -939,8 +915,64 @@
<td>Default Null value representation in the data being loaded. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#carbon.options.serialization.null.format for detailed information.</td>
</tr>
<tr>
-<td>carbon.query.directQueryOnDataMap.enabled</td>
-<td>Specifies whether datamap can be queried directly. This is useful for debugging purposes.**NOTE: **Refer to <a href="#query-configuration">Query Configuration</a> for detailed information.</td>
+<td>carbon.number.of.cores.while.loading</td>
+<td>Specifies number of cores to be used while loading data. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#carbon.number.of.cores.while.loading for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.number.of.cores.while.compacting</td>
+<td>Specifies number of cores to be used while compacting data. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#carbon.number.of.cores.while.compacting for detailed information.</td>
+</tr>
+<tr>
+<td>enable.offheap.sort</td>
+<td>To enable off-heap memory usage. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#enable.offheap.sort for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.blockletgroup.size.in.mb</td>
+<td>Specifies the size of each blocklet group. <strong>NOTE:</strong> Refer to <a href="#data-loading-configuration">Data Loading Configuration</a>#carbon.blockletgroup.size.in.mb for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.enable.auto.load.merge</td>
+<td>To enable compaction along with data loading. <strong>NOTE:</strong> Refer to <a href="#compaction-configuration">Compaction Configuration</a>#carbon.enable.auto.load.merge for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.major.compaction.size</td>
+<td>To configure major compaction size. <strong>NOTE:</strong> Refer to <a href="#compaction-configuration">Compaction Configuration</a>#carbon.major.compaction.size for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.compaction.level.threshold</td>
+<td>To configure compaction threshold. <strong>NOTE:</strong> Refer to <a href="#compaction-configuration">Compaction Configuration</a>#carbon.compaction.level.threshold for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.enable.vector.reader</td>
+<td>To enable fetching data as columnar batch of size 4*1024 rows instead of fetching a row at a time. <strong>NOTE:</strong> Refer to <a href="#query-configuration">Query Configuration</a>#carbon.enable.vector.reader for detailed information.</td>
+</tr>
+<tr>
+<td>enable.unsafe.in.query.processing</td>
+<td>To enable use of unsafe functions while scanning the data during query. <strong>NOTE:</strong> Refer to <a href="#query-configuration">Query Configuration</a>#enable.unsafe.in.query.processing for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.push.rowfilters.for.vector</td>
+<td>To enable complete row filters handling by carbon in case of vector. <strong>NOTE:</strong> Refer to <a href="#query-configuration">Query Configuration</a>#carbon.push.rowfilters.for.vector for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.query.stage.input.enable</td>
+<td>To make query to include staged input files. <strong>NOTE:</strong> Refer to <a href="#query-configuration">Query Configuration</a>#carbon.query.stage.input.enable for detailed information.</td>
+</tr>
+<tr>
+<td>carbon.input.segments.<db_name>.<table_name></td>
+<td>To specify the segment ids to query from the table. segments ids are separated by comma.</td>
+</tr>
+<tr>
+<td>carbon.index.visible.<db_name>.<table_name>.<index_name></td>
+<td>To specify query on <em><strong>db_name.table_name</strong></em> to not use the index <em><strong>index_name</strong></em>.</td>
+</tr>
+<tr>
+<td>carbon.load.indexes.parallel.<db_name>.<table_name></td>
+<td>To enable parallel index loading for a table. when db_name.table_name are not specified, i.e., when <em><strong>carbon.load.indexes.parallel.</strong></em> is set, it applies for all the tables of the session.</td>
+</tr>
+<tr>
+<td>carbon.enable.index.server</td>
+<td>To use index server for caching and pruning. This property can be used for a session or for a particular table with <em><strong>carbon.enable.index.server.<db_name>.<table_name></strong></em>.</td>
</tr>
</tbody>
</table>
diff --git a/content/csdk-guide.html b/content/csdk-guide.html
index d775939..7cecf10 100644
--- a/content/csdk-guide.html
+++ b/content/csdk-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -385,8 +394,7 @@ void outputPath(char *path);
*
* @param argc argc argument counter, the number of projection column
* @param argv argv is a string array of columns that needs to be sorted.
- * If it is null or by default all dimensions are selected for sorting
- * If it is empty array, no columns are sorted
+ * If it is null or empty array, no columns are selected for sorting.
*/
void sortBy(int argc, char *argv[]);
</code></pre>
diff --git a/content/ddl-of-carbondata.html b/content/ddl-of-carbondata.html
index 283fd3b..e9003be 100644
--- a/content/ddl-of-carbondata.html
+++ b/content/ddl-of-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -237,7 +246,6 @@
<li>
<p><a href="#create-table">CREATE TABLE</a></p>
<ul>
-<li><a href="#dictionary-encoding-configuration">Dictionary Encoding</a></li>
<li><a href="#local-dictionary-configuration">Local Dictionary</a></li>
<li><a href="#inverted-index-configuration">Inverted Index</a></li>
<li><a href="#sort-columns-configuration">Sort Columns</a></li>
@@ -248,11 +256,12 @@
<li><a href="#caching-minmax-value-for-required-columns">Caching Column Min/Max</a></li>
<li><a href="#caching-at-block-or-blocklet-level">Caching Level</a></li>
<li><a href="#support-flat-folder-same-as-hiveparquet">Hive/Parquet folder Structure</a></li>
-<li><a href="#string-longer-than-32000-characters">Extra Long String columns</a></li>
+<li><a href="#string-longer-than-32000-characters">Long String columns</a></li>
<li><a href="#compression-for-table">Compression for Table</a></li>
<li><a href="#bad-records-path">Bad Records Path</a></li>
<li><a href="#load-minimum-data-size">Load Minimum Input File Size</a></li>
<li><a href="#range-column">Range Column</a></li>
+<li><a href="#index-cache-expiration-time-in-seconds">Index Cache Expiration Time In Seconds</a></li>
</ul>
</li>
<li>
@@ -298,14 +307,6 @@
<li><a href="#insert-overwrite">INSERT OVERWRITE PARTITION</a></li>
</ul>
</li>
-<li>
-<a href="#create-hash-partition-table">CARBONDATA PARTITION</a>
-<ul>
-<li><a href="#create-hash-partition-table">HASH PARTITION</a></li>
-<li><a href="#create-range-partition-table">RANGE PARTITION</a></li>
-<li><a href="#create-list-partition-table">LIST PARTITION</a></li>
-</ul>
-</li>
<li><a href="#show-partitions">SHOW PARTITIONS</a></li>
<li><a href="#add-a-new-partition">ADD PARTITION</a></li>
<li><a href="#split-a-partition">SPLIT PARTITION</a></li>
@@ -327,7 +328,7 @@ STORED AS carbondata
[TBLPROPERTIES (property_name=property_value, ...)]
[LOCATION 'path']
</code></pre>
-<p><strong>NOTE:</strong> CarbonData also supports "STORED AS carbondata" and "USING carbondata". Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala" target=_blank>CarbonSessionExample</a> in the CarbonData repo.</p>
+<p><strong>NOTE:</strong> CarbonData also supports "STORED AS carbondata" and "USING carbondata". Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala" target=_blank>CarbonSessionExample</a> in the CarbonData repo.</p>
<h3>
<a id="usage-guidelines" class="anchor" href="#usage-guidelines" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage Guidelines</h3>
<p><strong>Supported properties:</strong></p>
@@ -340,10 +341,6 @@ STORED AS carbondata
</thead>
<tbody>
<tr>
-<td><a href="#dictionary-encoding-configuration">DICTIONARY_INCLUDE</a></td>
-<td>Columns for which dictionary needs to be generated</td>
-</tr>
-<tr>
<td><a href="#inverted-index-configuration">NO_INVERTED_INDEX</a></td>
<td>Columns to exclude from inverted index generation</td>
</tr>
@@ -428,11 +425,11 @@ STORED AS carbondata
<td>Columns which are greater than 32K characters</td>
</tr>
<tr>
-<td><a href="#bucketing">BUCKETNUMBER</a></td>
+<td><a href="#bucketing">BUCKET_NUMBER</a></td>
<td>Number of buckets to be created</td>
</tr>
<tr>
-<td><a href="#bucketing">BUCKETCOLUMNS</a></td>
+<td><a href="#bucketing">BUCKET_COLUMNS</a></td>
<td>Columns which are to be placed in buckets</td>
</tr>
<tr>
@@ -443,29 +440,20 @@ STORED AS carbondata
<td><a href="#range-column">Range Column</a></td>
<td>partition input data by range</td>
</tr>
+<tr>
+<td><a href="#index-cache-expiration-time-in-seconds">INDEX_CACHE_EXPIRATION_TIME_IN_SECONDS</a></td>
+<td>Table level time-based cache expiration in seconds</td>
+</tr>
</tbody>
</table>
<p>Following are the guidelines for TBLPROPERTIES, CarbonData's additional table options can be set via carbon.properties.</p>
<ul>
<li>
<h5>
-<a id="dictionary-encoding-configuration" class="anchor" href="#dictionary-encoding-configuration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Dictionary Encoding Configuration</h5>
-<p>Dictionary encoding is turned off for all columns by default from 1.3 onwards, you can use this command for including or excluding columns to do dictionary encoding.
-Suggested use cases : do dictionary encoding for low cardinality columns, it might help to improve data compression ratio and performance.</p>
-<pre><code>TBLPROPERTIES ('DICTIONARY_INCLUDE'='column1, column2')
-</code></pre>
-<p><strong>NOTE</strong>:</p>
-<ul>
-<li>Dictionary Include/Exclude for complex child columns is not supported. Dictionary Include doesn't support binary data type.</li>
-<li>Dictionary is global. Except global dictionary, there are local dictionary and non-dictionary in CarbonData.</li>
-</ul>
-</li>
-<li>
-<h5>
<a id="local-dictionary-configuration" class="anchor" href="#local-dictionary-configuration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Local Dictionary Configuration</h5>
</li>
</ul>
-<p>Columns for which dictionary is not generated needs more storage space and in turn more IO. Also since more data will have to be read during query, query performance also would suffer.Generating dictionary per blocklet for such columns would help in saving storage space and assist in improving query performance as carbondata is optimized for handling dictionary encoded columns more effectively.Generating dictionary internally per blocklet is termed as local dictionary. Please refer to [...]
+<p>Columns for which dictionary is not generated needs more storage space and in turn more IO. Also since more data will have to be read during query, query performance also would suffer. Generating dictionary per blocklet for such columns would help in saving storage space and assist in improving query performance as carbondata is optimized for handling dictionary encoded columns more effectively.Generating dictionary internally per blocklet is termed as local dictionary. Please refer t [...]
<p>Local Dictionary helps in:</p>
<ol>
<li>Getting more compression.</li>
@@ -548,7 +536,7 @@ Suggested use cases : do dictionary encoding for low cardinality columns, it mig
<tr>
<td>LOCAL_DICTIONARY_INCLUDE</td>
<td>string/varchar/char columns</td>
-<td>Columns for which Local Dictionary has to be generated.<strong>NOTE:</strong> Those string/varchar/char columns which are added into DICTIONARY_INCLUDE option will not be considered for local dictionary generation. This property needs to be configured only when local dictionary needs to be generated for few columns, skipping others. This property takes effect only when <strong>LOCAL_DICTIONARY_ENABLE</strong> is true or <strong>carbon.local.dictionary.enable</strong> is true</td>
+<td>Columns for which Local Dictionary has to be generated. This property needs to be configured only when local dictionary needs to be generated for few columns, skipping others. This property takes effect only when <strong>LOCAL_DICTIONARY_ENABLE</strong> is true or <strong>carbon.local.dictionary.enable</strong> is true</td>
</tr>
<tr>
<td>LOCAL_DICTIONARY_EXCLUDE</td>
@@ -587,9 +575,6 @@ TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='true','LOCAL_DICTIONARY_THRESHOLD'='100
<p><strong>NOTE:</strong></p>
<ul>
<li>We recommend to use Local Dictionary when cardinality is high but is distributed across multiple loads</li>
-<li>On a large cluster, decoding data can become a bottleneck for global dictionary as there will be many remote reads. In this scenario, it is better to use Local Dictionary.</li>
-<li>When cardinality is less, but loads are repetitive, it is better to use global dictionary as local dictionary generates multiple dictionary files at blocklet level increasing redundancy.</li>
-<li>If want to use non-dictionary, users can set LOCAL_DICTIONARY_ENABLE as false and don't set DICTIONARY_INCLUDE.</li>
</ul>
<ul>
<li>
@@ -598,7 +583,7 @@ TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='true','LOCAL_DICTIONARY_THRESHOLD'='100
<p>By default inverted index is disabled as store size will be reduced, it can be enabled by using a table property. It might help to improve compression ratio and query speed, especially for low cardinality columns which are in reward position.
Suggested use cases : For high cardinality columns, you can disable the inverted index for improving the data loading performance.</p>
<p><strong>NOTE</strong>: Columns specified in INVERTED_INDEX should also be present in SORT_COLUMNS.</p>
-<pre><code>TBLPROPERTIES ('SORT_COLUMNS'='column2,column3','NO_INVERTED_INDEX'='column1', 'INVERTED_INDEX'='column2, column3')
+<pre><code>TBLPROPERTIES ('SORT_COLUMNS'='column2,column3', 'INVERTED_INDEX'='column2, column3')
</code></pre>
</li>
<li>
@@ -613,7 +598,7 @@ Suggested use cases : Only build MDK index for required columns,it might help to
</ul>
<pre><code>TBLPROPERTIES ('SORT_COLUMNS'='column1, column3')
</code></pre>
-<p><strong>NOTE</strong>: Sort_Columns for Complex datatype columns and binary data type is not supported.</p>
+<p><strong>NOTE</strong>: Sort_Columns for Complex datatype columns, binary, double, float, decimal data type is not supported.</p>
</li>
<li>
<h5>
@@ -622,7 +607,6 @@ Suggested use cases : Only build MDK index for required columns,it might help to
<ul>
<li>LOCAL_SORT: data will be locally sorted (task level sorting)</li>
<li>NO_SORT: default scope. It will load the data in unsorted manner, it will significantly increase load performance.</li>
-<li>BATCH_SORT: It increases the load performance but decreases the query performance if identified blocks > parallelism.</li>
<li>GLOBAL_SORT: It increases the query performance, especially high concurrent point query.
And if you care about loading resources isolation strictly, because the system uses the spark GroupBy to sort data, the resource can be controlled by spark.</li>
</ul>
@@ -641,9 +625,9 @@ And if you care about loading resources isolation strictly, because the system u
revenue INT)
STORED AS carbondata
TBLPROPERTIES ('SORT_COLUMNS'='productName,storeCity',
- 'SORT_SCOPE'='NO_SORT')
+ 'SORT_SCOPE'='LOCAL_SORT')
</code></pre>
-<p><strong>NOTE:</strong> CarbonData also supports "using carbondata". Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/SparkSessionExample.scala" target=_blank>SparkSessionExample</a> in the CarbonData repo.</p>
+<p><strong>NOTE:</strong> CarbonData also supports "using carbondata". Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/scala/org/apache/carbondata/examples/SparkSessionExample.scala" target=_blank>SparkSessionExample</a> in the CarbonData repo.</p>
<ul>
<li>
<h5>
@@ -726,10 +710,10 @@ Following are 5 configurations:</p>
<p>Columns to be cached can be specified either while creating table or after creation of the table.
During create table operation; specify the columns to be cached in table properties.</p>
<p>Syntax:</p>
-<pre><code>CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,?) STORED BY 'carbondata' TBLPROPERTIES ('COLUMN_META_CACHE'='col1,col2,?')
+<pre><code>CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,?) STORED AS carbondata TBLPROPERTIES ('COLUMN_META_CACHE'='col1,col2,?')
</code></pre>
<p>Example:</p>
-<pre><code>CREATE TABLE employee (name String, city String, id int) STORED BY 'carbondata' TBLPROPERTIES ('COLUMN_META_CACHE'='name')
+<pre><code>CREATE TABLE employee (name String, city String, id int) STORED AS carbondata TBLPROPERTIES ('COLUMN_META_CACHE'='name')
</code></pre>
<p>After creation of table or on already created tables use the alter table command to configure the columns to be cached.</p>
<p>Syntax:</p>
@@ -753,10 +737,10 @@ During create table operation; specify the columns to be cached in table propert
<p>Cache level can be specified either while creating table or after creation of the table.
During create table operation specify the cache level in table properties.</p>
<p>Syntax:</p>
-<pre><code>CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,?) STORED BY 'carbondata' TBLPROPERTIES ('CACHE_LEVEL'='Blocklet')
+<pre><code>CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,?) STORED AS carbondata TBLPROPERTIES ('CACHE_LEVEL'='Blocklet')
</code></pre>
<p>Example:</p>
-<pre><code>CREATE TABLE employee (name String, city String, id int) STORED BY 'carbondata' TBLPROPERTIES ('CACHE_LEVEL'='Blocklet')
+<pre><code>CREATE TABLE employee (name String, city String, id int) STORED AS carbondata TBLPROPERTIES ('CACHE_LEVEL'='Blocklet')
</code></pre>
<p>After creation of table or on already created tables use the alter table command to configure the cache level.</p>
<p>Syntax:</p>
@@ -774,7 +758,7 @@ During create table operation specify the cache level in table properties.</p>
<pre><code>'flat_folder'='true'
</code></pre>
<p>Example:</p>
-<pre><code>CREATE TABLE employee (name String, city String, id int) STORED BY 'carbondata' TBLPROPERTIES ('flat_folder'='true')
+<pre><code>CREATE TABLE employee (name String, city String, id int) STORED AS carbondata TBLPROPERTIES ('flat_folder'='true')
</code></pre>
</li>
<li>
@@ -796,15 +780,15 @@ TBLPROPERTIES ('LONG_STRING_COLUMNS'='col1,col2')
</code></pre>
<p>If you are using Carbon-SDK, you can specify the datatype of long string column as <code>varchar</code>.
You can refer to SDKwriterTestCase for example.</p>
-<p><strong>NOTE:</strong> The LONG_STRING_COLUMNS can only be string/char/varchar columns and cannot be dictionary_include/sort_columns/complex columns.</p>
+<p><strong>NOTE:</strong> The LONG_STRING_COLUMNS can only be string/char/varchar columns and cannot be sort_columns/complex columns.</p>
</li>
<li>
<h5>
<a id="compression-for-table" class="anchor" href="#compression-for-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compression for table</h5>
<p>Data compression is also supported by CarbonData.
-By default, Snappy is used to compress the data. CarbonData also supports ZSTD compressor.
-User can specify the compressor in the table property:</p>
-<pre><code>TBLPROPERTIES('carbon.column.compressor'='snappy')
+By default, Snappy is used to compress the data. CarbonData also supports ZSTD and GZIP compressors.</p>
+<p>User can specify the compressor in the table property:</p>
+<pre><code>TBLPROPERTIES('carbon.column.compressor'='GZIP')
</code></pre>
<p>or</p>
<pre><code>TBLPROPERTIES('carbon.column.compressor'='zstd')
@@ -850,6 +834,18 @@ This feature doesn't support binary data type.</p>
<pre><code>TBLPROPERTIES('RANGE_COLUMN'='col1')
</code></pre>
</li>
+<li>
+<h5>
+<a id="index-cache-expiration-time-in-seconds" class="anchor" href="#index-cache-expiration-time-in-seconds" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Cache Expiration Time In Seconds</h5>
+<p>Carbon maintains index cache in driver side and the cache will be expired after seconds indicated by this table property.</p>
+<pre><code>TBLPROPERTIES('index_cache_expiration_seconds'='1')
+</code></pre>
+<p>After creation of table or on already created tables use the alter table command to configure the cache expiration time.</p>
+<p>Syntax:</p>
+<pre><code> ALTER TABLE [dbName].tableName SET TBLPROPERTIES ('index_cache_expiration_seconds'='3')
+
+</code></pre>
+</li>
</ul>
<h2>
<a id="create-table-as-select" class="anchor" href="#create-table-as-select" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CREATE TABLE AS SELECT</h2>
@@ -916,7 +912,7 @@ sql(
| STORED AS carbondata
| LOCATION '$storeLocation/origin'
""".stripMargin)
-checkAnswer(sql("SELECT count(*) from source"), sql("SELECT count(*) from origin"))
+sql("SELECT count(*) from source").show()
</code></pre>
<h3>
<a id="create-external-table-on-non-transactional-table-data-location" class="anchor" href="#create-external-table-on-non-transactional-table-data-location" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create external table on Non-Transactional table data location.</h3>
@@ -933,12 +929,10 @@ Our SDK module currently supports writing data in this format.</p>
This can be SDK output or C++ SDK output. Refer <a href="./sdk-guide.html">SDK Guide</a> and <a href="./csdk-guide.html">C++ SDK Guide</a>.</p>
<p><strong>Note:</strong></p>
<ol>
-<li>Dropping of the external table should not delete the files present in the location.</li>
+<li>Dropping of the external table will not delete the files present in the location.</li>
<li>When external table is created on non-transactional table data,
external table will be registered with the schema of carbondata files.
-If multiple files with different schema is present, exception will be thrown.
-So, If table registered with one schema and files are of different schema,
-suggest to drop the external table and create again to register table with new schema.</li>
+If multiple files have the same column with different datatypes then exception will be thrown.</li>
</ol>
<h2>
<a id="create-database" class="anchor" href="#create-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CREATE DATABASE</h2>
@@ -982,21 +976,21 @@ ALTER TABLE test_db.carbon RENAME TO test_db.carbonTable
<a id="add-columns" class="anchor" href="#add-columns" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>ADD COLUMNS</h4>
<p>This command is used to add a new column to the existing table.</p>
<pre><code>ALTER TABLE [db_name.]table_name ADD COLUMNS (col_name data_type,...)
-TBLPROPERTIES('DICTIONARY_INCLUDE'='col_name,...',
-'DEFAULT.VALUE.COLUMN_NAME'='default_value')
+TBLPROPERTIES('DEFAULT.VALUE.COLUMN_NAME'='default_value')
</code></pre>
<p>Examples:</p>
<pre><code>ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING)
</code></pre>
-<pre><code>ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING) TBLPROPERTIES('DICTIONARY_INCLUDE'='a1')
-</code></pre>
<pre><code>ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING) TBLPROPERTIES('DEFAULT.VALUE.a1'='10')
</code></pre>
<p><strong>NOTE:</strong> Add Complex datatype columns is not supported.</p>
</li>
</ul>
-<p>Users can specify which columns to include and exclude for local dictionary generation after adding new columns. These will be appended with the already existing local dictionary include and exclude columns of main table respectively.
-<code>ALTER TABLE carbon ADD COLUMNS (a1 STRING, b1 STRING) TBLPROPERTIES('LOCAL_DICTIONARY_INCLUDE'='a1','LOCAL_DICTIONARY_EXCLUDE'='b1')</code></p>
+<p>Users can specify which columns to include and exclude for local dictionary generation after adding new columns. These will be appended with the already existing local dictionary include and exclude columns of main table respectively.</p>
+<pre><code> ```
+ ALTER TABLE carbon ADD COLUMNS (a1 STRING, b1 STRING) TBLPROPERTIES('LOCAL_DICTIONARY_INCLUDE'='a1','LOCAL_DICTIONARY_EXCLUDE'='b1')
+ ```
+</code></pre>
<ul>
<li>
<h4>
@@ -1144,7 +1138,6 @@ STORED AS carbondata
productNumber Int COMMENT 'unique serial number for product')
COMMENT "This is table comment"
STORED AS carbondata
-TBLPROPERTIES ('DICTIONARY_INCLUDE'='productNumber')
</code></pre>
<p>You can also SET and UNSET table comment using ALTER command.</p>
<p>Example to SET table comment:</p>
@@ -1165,7 +1158,7 @@ TBLPROPERTIES ('DICTIONARY_INCLUDE'='productNumber')
[(col_name data_type , ...)]
[COMMENT table_comment]
[PARTITIONED BY (col_name data_type , ...)]
- [STORED BY file_format]
+ [STORED AS file_format]
[TBLPROPERTIES (property_name=property_value, ...)]
</code></pre>
<p>Example:</p>
@@ -1207,80 +1200,6 @@ SELECT * FROM another_user au
WHERE au.country = 'US';
</code></pre>
<h3>
-<a id="carbondata-partitionhashrangelist----alpha-feature-this-partition-feature-does-not-support-update-and-delete-data" class="anchor" href="#carbondata-partitionhashrangelist----alpha-feature-this-partition-feature-does-not-support-update-and-delete-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CARBONDATA PARTITION(HASH,RANGE,LIST) -- Alpha feature, this partition feature does not support update and delete data.</h3>
-<p>The partition supports three type:(Hash,Range,List), similar to other system's partition features, CarbonData's partition feature can be used to improve query performance by filtering on the partition column.</p>
-<h3>
-<a id="create-hash-partition-table" class="anchor" href="#create-hash-partition-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Hash Partition Table</h3>
-<p>This command allows us to create hash partition.</p>
-<pre><code>CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- [(col_name data_type , ...)]
-PARTITIONED BY (partition_col_name data_type)
-STORED AS carbondata
-[TBLPROPERTIES ('PARTITION_TYPE'='HASH',
- 'NUM_PARTITIONS'='N' ...)]
-</code></pre>
-<p><strong>NOTE:</strong> N is the number of hash partitions</p>
-<p>Example:</p>
-<pre><code>CREATE TABLE IF NOT EXISTS hash_partition_table(
- col_A STRING,
- col_B INT,
- col_C LONG,
- col_D DECIMAL(10,2),
- col_F TIMESTAMP
-) PARTITIONED BY (col_E LONG)
-STORED AS carbondata TBLPROPERTIES('PARTITION_TYPE'='HASH','NUM_PARTITIONS'='9')
-</code></pre>
-<h3>
-<a id="create-range-partition-table" class="anchor" href="#create-range-partition-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Range Partition Table</h3>
-<p>This command allows us to create range partition.</p>
-<pre><code>CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- [(col_name data_type , ...)]
-PARTITIONED BY (partition_col_name data_type)
-STORED AS carbondata
-[TBLPROPERTIES ('PARTITION_TYPE'='RANGE',
- 'RANGE_INFO'='2014-01-01, 2015-01-01, 2016-01-01, ...')]
-</code></pre>
-<p><strong>NOTE:</strong></p>
-<ul>
-<li>The 'RANGE_INFO' must be defined in ascending order in the table properties.</li>
-<li>The default format for partition column of Date/Timestamp type is yyyy-MM-dd. Alternate formats for Date/Timestamp could be defined in CarbonProperties.</li>
-</ul>
-<p>Example:</p>
-<pre><code>CREATE TABLE IF NOT EXISTS range_partition_table(
- col_A STRING,
- col_B INT,
- col_C LONG,
- col_D DECIMAL(10,2),
- col_E LONG
-) partitioned by (col_F Timestamp)
-STORED BY 'carbondata'
-TBLPROPERTIES('PARTITION_TYPE'='RANGE',
-'RANGE_INFO'='2015-01-01, 2016-01-01, 2017-01-01, 2017-02-01')
-</code></pre>
-<h3>
-<a id="create-list-partition-table" class="anchor" href="#create-list-partition-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create List Partition Table</h3>
-<p>This command allows us to create list partition.</p>
-<pre><code>CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- [(col_name data_type , ...)]
-PARTITIONED BY (partition_col_name data_type)
-STORED AS carbondata
-[TBLPROPERTIES ('PARTITION_TYPE'='LIST',
- 'LIST_INFO'='A, B, C, ...')]
-</code></pre>
-<p><strong>NOTE:</strong> List partition supports list info in one level group.</p>
-<p>Example:</p>
-<pre><code>CREATE TABLE IF NOT EXISTS list_partition_table(
- col_B INT,
- col_C LONG,
- col_D DECIMAL(10,2),
- col_E LONG,
- col_F TIMESTAMP
- ) PARTITIONED BY (col_A STRING)
-STORED AS carbondata
-TBLPROPERTIES('PARTITION_TYPE'='LIST',
-'LIST_INFO'='aaaa, bbbb, (cccc, dddd), eeee')
-</code></pre>
-<h3>
<a id="show-partitions-1" class="anchor" href="#show-partitions-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Show Partitions</h3>
<p>The following command is executed to get the partition information of the table</p>
<pre><code>SHOW PARTITIONS [db_name.]table_name
@@ -1325,8 +1244,8 @@ of columns is used.</p>
<pre><code>CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type, ...)]
STORED AS carbondata
-TBLPROPERTIES('BUCKETNUMBER'='noOfBuckets',
-'BUCKETCOLUMNS'='columnname')
+TBLPROPERTIES('BUCKET_NUMBER'='noOfBuckets',
+'BUCKET_COLUMNS'='columnname')
</code></pre>
<p><strong>NOTE:</strong></p>
<ul>
@@ -1344,7 +1263,7 @@ TBLPROPERTIES('BUCKETNUMBER'='noOfBuckets',
productBatch STRING,
revenue INT)
STORED AS carbondata
-TBLPROPERTIES ('BUCKETNUMBER'='4', 'BUCKETCOLUMNS'='productName')
+TBLPROPERTIES ('BUCKET_NUMBER'='4', 'BUCKET_COLUMNS'='productName')
</code></pre>
<h2>
<a id="cache" class="anchor" href="#cache" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CACHE</h2>
@@ -1352,15 +1271,19 @@ TBLPROPERTIES ('BUCKETNUMBER'='4', 'BUCKETCOLUMNS'='productName')
about current cache used status in memory through the following command:</p>
<div class="highlight highlight-source-sql"><pre>SHOW METACACHE</pre></div>
<p>This shows the overall memory consumed in the cache by categories - index files, dictionary and
-datamaps. This also shows the cache usage by all the tables and children tables in the current
+indexes. This also shows the cache usage by all the tables and children tables in the current
database.</p>
+<div class="highlight highlight-source-sql"><pre> SHOW EXECUTOR METACACHE</pre></div>
+<p>This shows the overall memory consumed by the cache in each executor of the Index
+Server. This command is only allowed when the carbon property <code>carbon.enable.index.server</code>
+is set to true.</p>
<div class="highlight highlight-source-sql"><pre>SHOW METACACHE <span class="pl-k">ON</span> TABLE tableName</pre></div>
<p>This shows detailed information on cache usage by the table <code>tableName</code> and its carbonindex files,
-its dictionary files, its datamaps and children tables.</p>
+its dictionary files, its indexes and children tables.</p>
<p>This command is not allowed on child tables.</p>
<div class="highlight highlight-source-sql"><pre> DROP METACACHE <span class="pl-k">ON</span> TABLE tableName</pre></div>
<p>This clears any entry in cache by the table <code>tableName</code>, its carbonindex files,
-its dictionary files, its datamaps and children tables.</p>
+its dictionary files, its indexes and children tables.</p>
<p>This command is not allowed on child tables.</p>
<h3>
<a id="important-points" class="anchor" href="#important-points" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Important points</h3>
diff --git a/content/dml-of-carbondata.html b/content/dml-of-carbondata.html
index 0adce3b..2e85079 100644
--- a/content/dml-of-carbondata.html
+++ b/content/dml-of-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -236,6 +245,7 @@
<ul>
<li><a href="#load-data">LOAD DATA</a></li>
<li><a href="#insert-data-into-carbondata-table">INSERT DATA</a></li>
+<li><a href="#insert-data-into-carbondata-table-from-stage-input-files">INSERT DATA STAGE</a></li>
<li><a href="#load-data-using-static-partition">Load Data Using Static Partition</a></li>
<li><a href="#load-data-using-dynamic-partition">Load Data Using Dynamic Partition</a></li>
<li><a href="#update-and-delete">UPDATE AND DELETE</a></li>
@@ -272,6 +282,10 @@ OPTIONS(property_name=property_value, ...)
<td>Character used to quote the data in the input csv file</td>
</tr>
<tr>
+<td><a href="#line_separator">LINE_SEPARATOR</a></td>
+<td>Characters used to specify the line separator in the input csv file. If not provided, csv parser will detect it automatically.</td>
+</tr>
+<tr>
<td><a href="#commentchar">COMMENTCHAR</a></td>
<td>Character used to comment the rows in the input csv file. Those rows will be skipped from processing</td>
</tr>
@@ -308,12 +322,8 @@ OPTIONS(property_name=property_value, ...)
<td>Ending delimiter for complex type data in input csv file</td>
</tr>
<tr>
-<td><a href="#all_dictionary_path">ALL_DICTIONARY_PATH</a></td>
-<td>Path to read the dictionary data from all columns</td>
-</tr>
-<tr>
-<td><a href="#columndict">COLUMNDICT</a></td>
-<td>Path to read the dictionary data from for particular column</td>
+<td><a href="#complex_delimiter_level_3">COMPLEX_DELIMITER_LEVEL_3</a></td>
+<td>Ending delimiter for nested complex type data in input csv file of level 3.</td>
</tr>
<tr>
<td><a href="#dateformattimestampformat">DATEFORMAT</a></td>
@@ -328,10 +338,6 @@ OPTIONS(property_name=property_value, ...)
<td>How to partition the sort columns to make the evenly distributed</td>
</tr>
<tr>
-<td><a href="#single_pass">SINGLE_PASS</a></td>
-<td>When to enable single pass data loading</td>
-</tr>
-<tr>
<td><a href="#bad-records-handling">BAD_RECORDS_LOGGER_ENABLE</a></td>
<td>Whether to enable bad records logging</td>
</tr>
@@ -355,10 +361,6 @@ OPTIONS(property_name=property_value, ...)
<td><a href="#scale_factor">SCALE_FACTOR</a></td>
<td>Control the partition size for RANGE_COLUMN feature</td>
</tr>
-<tr>
-<td>[CARBON_OPTIONS_BINARY_DECODER]</td>
-<td>Support configurable decode for loading from csv</td>
-</tr>
</tbody>
</table>
<ul>
@@ -381,6 +383,13 @@ OPTIONS(property_name=property_value, ...)
</li>
<li>
<h5>
+<a id="line_separator" class="anchor" href="#line_separator" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>LINE_SEPARATOR:</h5>
+<p>Line separator Characters can be provided in the load command.</p>
+<pre><code>OPTIONS('LINE_SEPARATOR'='\n')
+</code></pre>
+</li>
+<li>
+<h5>
<a id="commentchar" class="anchor" href="#commentchar" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>COMMENTCHAR:</h5>
<p>Comment Characters can be provided in the load command if user wants to comment lines.</p>
<pre><code>OPTIONS('COMMENTCHAR'='#')
@@ -408,19 +417,19 @@ true: CSV file is with file header.</p>
<a id="sort_scope" class="anchor" href="#sort_scope" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SORT_SCOPE:</h5>
<p>Sort Scope to be used for the current load. This overrides the Sort Scope of Table.
Requirement: Sort Columns must be set while creating table. If Sort Columns is null, Sort Scope is always NO_SORT.</p>
-<pre><code>OPTIONS('SORT_SCOPE'='BATCH_SORT')
+<pre><code>OPTIONS('SORT_SCOPE'='GLOBAL_SORT')
</code></pre>
<p>Priority order for choosing Sort Scope is:</p>
-<ol>
+<ul>
<li>Load Data Command</li>
-<li>CARBON.TABLE.LOAD.SORT.SCOPE..<table> session property
-</table>
-</li>
+<li>
+<code>CARBON.TABLE.LOAD.SORT.SCOPE.<db>.<table></code> session property.</li>
<li>Table level Sort Scope</li>
-<li>CARBON.OPTIONS.SORT.SCOPE session property</li>
+<li>
+<code>CARBON.OPTIONS.SORT.SCOPE</code> session property</li>
<li>Default Value: NO_SORT</li>
-
-
+</ul>
+</li>
<li>
<h5>
<a id="multiline" class="anchor" href="#multiline" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MULTILINE:</h5>
@@ -466,26 +475,11 @@ Used in case of nested Complex Map type. (eg., 'a\003b\002b\003c\001aa\003bb\002
</li>
<li>
<h5>
-<a id="all_dictionary_path" class="anchor" href="#all_dictionary_path" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>ALL_DICTIONARY_PATH:</h5>
-<p>All dictionary files path.</p>
-<pre><code>OPTIONS('ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary')
-</code></pre>
-</li>
-<li>
-<h5>
-<a id="columndict" class="anchor" href="#columndict" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>COLUMNDICT:</h5>
-<p>Dictionary file path for specified column.</p>
-<pre><code>OPTIONS('COLUMNDICT'='column1:dictionaryFilePath1,column2:dictionaryFilePath2')
-</code></pre>
-<p><strong>NOTE:</strong> ALL_DICTIONARY_PATH and COLUMNDICT can't be used together.</p>
-</li>
-<li>
-<h5>
<a id="dateformattimestampformat" class="anchor" href="#dateformattimestampformat" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DATEFORMAT/TIMESTAMPFORMAT:</h5>
<p>Date and Timestamp format for specified column.</p>
<pre><code>OPTIONS('DATEFORMAT' = 'yyyy-MM-dd','TIMESTAMPFORMAT'='yyyy-MM-dd HH:mm:ss')
</code></pre>
-<p><strong>NOTE:</strong> Date formats are specified by date pattern strings. The date pattern letters in CarbonData are same as in JAVA. Refer to <a href="http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html" target=_blank rel="nofollow">SimpleDateFormat</a>.</p>
+<p><strong>NOTE:</strong> Date formats are specified by date pattern strings. The date pattern in CarbonData is the same as in JAVA. Refer to <a href="http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html" target=_blank rel="nofollow">SimpleDateFormat</a>.</p>
</li>
<li>
<h5>
@@ -499,41 +493,12 @@ Used in case of nested Complex Map type. (eg., 'a\003b\002b\003c\001aa\003bb\002
<ul>
<li>SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'.</li>
<li>Carbondata will use these bounds as ranges to process data concurrently during the final sort procedure. The records will be sorted and written out inside each partition. Since the partition is sorted, all records will be sorted.</li>
-<li>Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'.</li>
<li>The option works better if your CPU usage during loading is low. If your current system CPU usage is high, better not to use this option. Besides, it depends on the user to specify the bounds. If user does not know the exactly bounds to make the data distributed evenly among the bounds, loading performance will still be better than before or at least the same as before.</li>
-<li>Users can find more information about this option in the description of PR1953.</li>
+<li>Users can find more information about this option in the description of <a href="https://github.com/apache/carbondata/pull/1953" target=_blank>PR1953</a>.</li>
</ul>
</li>
<li>
<h5>
-<a id="single_pass" class="anchor" href="#single_pass" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SINGLE_PASS:</h5>
-<p>Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary.</p>
-</li>
-
-<p>This option specifies whether to use single pass for loading data or not. By default this option is set to FALSE.</p>
-<pre><code> OPTIONS('SINGLE_PASS'='TRUE')
-</code></pre>
-<p><strong>NOTE:</strong></p>
-</ol>
-<ul>
-<li>If this option is set to TRUE then data loading will take less time.</li>
-<li>If this option is set to some invalid value other than TRUE or FALSE then it uses the default value.</li>
-</ul>
-<p>Example:</p>
-<pre><code>LOAD DATA inpath '/opt/rawdata/data.csv' INTO table carbontable
-options('DELIMITER'=',', 'QUOTECHAR'='"','COMMENTCHAR'='#',
-'HEADER'='false',
-'FILEHEADER'='empno,empname,designation,doj,workgroupcategory,
-workgroupcategoryname,deptno,deptname,projectcode,
-projectjoindate,projectenddate,attendance,utilization,salary',
-'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='\\\001',
-'COMPLEX_DELIMITER_LEVEL_2'='\\\002',
-'ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary',
-'SINGLE_PASS'='TRUE')
-</code></pre>
-<ul>
-<li>
-<h5>
<a id="bad-records-handling" class="anchor" href="#bad-records-handling" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BAD RECORDS HANDLING:</h5>
<p>Methods of handling bad records are as follows:</p>
<ul>
@@ -542,35 +507,31 @@ projectjoindate,projectenddate,attendance,utilization,salary',
</ul>
<pre><code>OPTIONS('BAD_RECORDS_LOGGER_ENABLE'='true', 'BAD_RECORD_PATH'='hdfs://hacluster/tmp/carbon', 'BAD_RECORDS_ACTION'='REDIRECT', 'IS_EMPTY_DATA_BAD_RECORD'='false')
</code></pre>
-</li>
-</ul>
<p><strong>NOTE:</strong></p>
<ul>
-<li>BAD_RECORDS_ACTION property can have four type of actions for bad records FORCE, REDIRECT, IGNORE and FAIL.</li>
+<li>BAD_RECORDS_ACTION property can have four types of actions for bad records FORCE, REDIRECT, IGNORE, and FAIL.</li>
<li>FAIL option is its Default value. If the FAIL option is used, then data loading fails if any bad records are found.</li>
-<li>If the REDIRECT option is used, CarbonData will add all bad records in to a separate CSV file. However, this file must not be used for subsequent data loading because the content may not exactly match the source record. You are advised to cleanse the original source record for further data ingestion. This option is used to remind you which records are bad records.</li>
+<li>If the REDIRECT option is used, CarbonData will add all bad records into a separate CSV file. However, this file must not be used for subsequent data loading because the content may not exactly match the source record. You are advised to cleanse the source record for further data ingestion. This option is used to remind you which records are bad.</li>
<li>If the FORCE option is used, then it auto-converts the data by storing the bad records as NULL before Loading data.</li>
<li>If the IGNORE option is used, then bad records are neither loaded nor written to the separate CSV file.</li>
<li>In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and the load operation fails.</li>
-<li>The default maximum number of characters per column is 32000. If there are more than 32000 characters in a column, please refer to <em>String longer than 32000 characters</em> section.</li>
+<li>The default maximum number of characters per column is 32000. If there are more than 32000 characters in a column, please refer to <a href="https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.html#string-longer-than-32000-characters">String longer than 32000 characters</a> section.</li>
<li>Since Bad Records Path can be specified in create, load and carbon properties.
-Therefore, value specified in load will have the highest priority, and value specified in carbon properties will have the least priority.</li>
+Therefore, the value specified in load will have the highest priority, and value specified in carbon properties will have the least priority.</li>
</ul>
<p>Example:</p>
<pre><code>LOAD DATA INPATH 'filepath.csv' INTO TABLE tablename
OPTIONS('BAD_RECORDS_LOGGER_ENABLE'='true','BAD_RECORD_PATH'='hdfs://hacluster/tmp/carbon',
'BAD_RECORDS_ACTION'='REDIRECT','IS_EMPTY_DATA_BAD_RECORD'='false')
</code></pre>
-<ul>
+</li>
<li>
<h5>
<a id="global_sort_partitions" class="anchor" href="#global_sort_partitions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>GLOBAL_SORT_PARTITIONS:</h5>
-<p>If the SORT_SCOPE is defined as GLOBAL_SORT, then user can specify the number of partitions to use while shuffling data for sort using GLOBAL_SORT_PARTITIONS. If it is not configured, or configured less than 1, then it uses the number of map task as reduce task. It is recommended that each reduce task deal with 512MB-1GB data.
+<p>If the SORT_SCOPE is defined as GLOBAL_SORT, then the user can specify the number of partitions to use while shuffling data for sort using GLOBAL_SORT_PARTITIONS. If it is not configured, or configured less than 1, then it uses the number of map tasks as reduce tasks. It is recommended that each reduce task deals with 512MB-1GB data.
For RANGE_COLUMN, GLOBAL_SORT_PARTITIONS is used to specify the number of range partitions also.
-GLOBAL_SORT_PARTITIONS should be specified optimally during RANGE_COLUMN LOAD because if a higher number is configured then the load time may be less but it will result in creation of more files which would degrade the query and compaction performance.
-Conversely, if less partitions are configured then the load performance may degrade due to less use of parallelism but the query and compaction will become faster. Hence the user may choose optimal number depending on the use case.</p>
-</li>
-</ul>
+GLOBAL_SORT_PARTITIONS should be specified optimally during RANGE_COLUMN LOAD because if a higher number is configured then the load time may be less but it will result in the creation of more files which would degrade the query and compaction performance.
+Conversely, if fewer partitions are configured then the load performance may degrade due to less use of parallelism but the query and compaction will become faster. Hence the user may choose an optimal number depending on the use case.</p>
<pre><code>OPTIONS('GLOBAL_SORT_PARTITIONS'='2')
</code></pre>
<p><strong>NOTE:</strong></p>
@@ -578,12 +539,10 @@ Conversely, if less partitions are configured then the load performance may degr
<li>GLOBAL_SORT_PARTITIONS should be Integer type, the range is [1,Integer.MaxValue].</li>
<li>It is only used when the SORT_SCOPE is GLOBAL_SORT.</li>
</ul>
-<ul>
+</li>
<li>
<h5>
<a id="scale_factor" class="anchor" href="#scale_factor" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SCALE_FACTOR</h5>
-</li>
-</ul>
<p>For RANGE_COLUMN, SCALE_FACTOR is used to control the number of range partitions as following.</p>
<pre><code> splitSize = max(blocklet_size, (block_size - blocklet_size)) * scale_factor
numPartitions = total size of input data / splitSize
@@ -596,25 +555,19 @@ Conversely, if less partitions are configured then the load performance may degr
<li>If both GLOBAL_SORT_PARTITIONS and SCALE_FACTOR are used at the same time, only GLOBAL_SORT_PARTITIONS is valid.</li>
<li>The compaction on RANGE_COLUMN will use LOCAL_SORT by default.</li>
</ul>
-<ul>
-<li>
-<h5>
-<a id="carbon_enable_range_compaction" class="anchor" href="#carbon_enable_range_compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CARBON_ENABLE_RANGE_COMPACTION</h5>
</li>
</ul>
-<p>To configure Ranges-based Compaction to be used or not for RANGE_COLUMN.
-The default value is 'true'.</p>
-
-
-<h3>
-<a id="insert-data-into-carbondata-table" class="anchor" href="#insert-data-into-carbondata-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>INSERT DATA INTO CARBONDATA TABLE</h3>
+</li>
+</ul>
+<h2>
+<a id="insert-data-into-carbondata-table" class="anchor" href="#insert-data-into-carbondata-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>INSERT DATA INTO CARBONDATA TABLE</h2>
<p>This command inserts data into a CarbonData table, it is defined as a combination of two queries Insert and Select query respectively.
It inserts records from a source table into a target CarbonData table, the source table can be a Hive table, Parquet table or a CarbonData table itself.
It comes with the functionality to aggregate the records of a table by performing Select query on source table and load its corresponding resultant records into a CarbonData table.</p>
<pre><code>INSERT INTO TABLE <CARBONDATA TABLE> SELECT * FROM sourceTableName
[ WHERE { <filter_condition> } ]
</code></pre>
-<p>You can also omit the <code>table</code> keyword and write your query as:</p>
+<p>User can also omit the <code>table</code> keyword and write the query as:</p>
<pre><code>INSERT INTO <CARBONDATA TABLE> SELECT * FROM sourceTableName
[ WHERE { <filter_condition> } ]
</code></pre>
@@ -636,8 +589,66 @@ It comes with the functionality to aggregate the records of a table by performin
</code></pre>
<pre><code>INSERT OVERWRITE TABLE table1 SELECT * FROM TABLE2
</code></pre>
-<h3>
-<a id="load-data-using-static-partition" class="anchor" href="#load-data-using-static-partition" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Load Data Using Static Partition</h3>
+<h2>
+<a id="insert-data-into-carbondata-table-from-stage-input-files" class="anchor" href="#insert-data-into-carbondata-table-from-stage-input-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>INSERT DATA INTO CARBONDATA TABLE From Stage Input Files</h2>
+<p>Stage input files are data files written by external application (such as Flink). These files
+are committed but not loaded into the table.</p>
+<p>User can use this command to insert them into the table, thus making them visible for a query.</p>
+<pre><code>INSERT INTO <CARBONDATA TABLE> STAGE OPTIONS(property_name=property_value, ...)
+</code></pre>
+<p><strong>Supported Properties:</strong></p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><a href="#batch_file_count">BATCH_FILE_COUNT</a></td>
+<td>The number of stage files per processing</td>
+</tr>
+<tr>
+<td><a href="#batch_file_order">BATCH_FILE_ORDER</a></td>
+<td>The order type of stage files in per processing</td>
+</tr>
+</tbody>
+</table>
+<ul>
+<li>
+<p>User can use the following options to load data:</p>
+<ul>
+<li>
+<h5>
+<a id="batch_file_count" class="anchor" href="#batch_file_count" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BATCH_FILE_COUNT:</h5>
+<p>The number of stage files per processing.</p>
+<pre><code>OPTIONS('batch_file_count'='5')
+</code></pre>
+</li>
+<li>
+<h5>
+<a id="batch_file_order" class="anchor" href="#batch_file_order" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BATCH_FILE_ORDER:</h5>
+<p>The order type of stage files in per processing, choices: ASC, DESC.
+The default is ASC.
+Stage files will order by the last modified time with the specified order type.</p>
+<pre><code>OPTIONS('batch_file_order'='DESC')
+</code></pre>
+<p>Examples:</p>
+<pre><code>INSERT INTO table1 STAGE
+
+INSERT INTO table1 STAGE OPTIONS('batch_file_count' = '5')
+Note: This command uses the default file order, will insert the earliest stage files into the table.
+
+INSERT INTO table1 STAGE OPTIONS('batch_file_count' = '5', 'batch_file_order'='DESC')
+Note: This command will insert the latest stage files into the table.
+</code></pre>
+</li>
+</ul>
+</li>
+</ul>
+<h2>
+<a id="load-data-using-static-partition" class="anchor" href="#load-data-using-static-partition" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Load Data Using Static Partition</h2>
<p>This command allows you to load data using static partition.</p>
<pre><code>LOAD DATA INPATH 'folder_path'
INTO TABLE [db_name.]table_name PARTITION (partition_spec)
@@ -654,8 +665,8 @@ INSERT INTO TABLE locationTable
PARTITION (country = 'US', state = 'AL')
SELECT <columns list excluding partition columns> FROM another_user
</code></pre>
-<h3>
-<a id="load-data-using-dynamic-partition" class="anchor" href="#load-data-using-dynamic-partition" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Load Data Using Dynamic Partition</h3>
+<h2>
+<a id="load-data-using-dynamic-partition" class="anchor" href="#load-data-using-dynamic-partition" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Load Data Using Dynamic Partition</h2>
<p>This command allows you to load data using dynamic partition. If partition spec is not specified, then the partition is considered as dynamic.</p>
<p>Example:</p>
<pre><code>LOAD DATA INPATH '${env:HOME}/staticinput.csv'
@@ -705,6 +716,45 @@ SET (column_name1, column_name2) =(select sourceColumn1, sourceColumn2 from sour
</code></pre>
<pre><code>DELETE FROM carbontable WHERE column1 IN (SELECT column11 FROM sourceTable2 WHERE column1 = 'USA')
</code></pre>
+<h3>
+<a id="delete-stage" class="anchor" href="#delete-stage" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DELETE STAGE</h3>
+<p>This command allows us to delete the data files (stage data) which is already loaded into the table.</p>
+<pre><code>DELETE FROM TABLE [db_name.]table_name STAGE OPTIONS(property_name=property_value, ...)
+</code></pre>
+<p><strong>Supported Properties:</strong></p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><a href="#retain_hour">retain_hour</a></td>
+<td>Data file retain time in hours</td>
+</tr>
+</tbody>
+</table>
+<ul>
+<li>
+<p>You can use the following options to delete data:</p>
+<ul>
+<li>
+<h5>
+<a id="retain_hour" class="anchor" href="#retain_hour" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>retain_hour:</h5>
+<p>Data file retain time in second, the command just delete overdue files only.</p>
+<pre><code>OPTIONS('retain_hour'='1')
+</code></pre>
+</li>
+</ul>
+<p>Examples:</p>
+<pre><code>DELETE FROM TABLE carbontable STAGE
+</code></pre>
+<pre><code>DELETE FROM TABLE carbontable STAGE OPTIONS ('retain_hour'='1')
+</code></pre>
+</li>
+</ul>
<h2>
<a id="compaction" class="anchor" href="#compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>COMPACTION</h2>
<p>Compaction improves the query performance significantly.</p>
@@ -714,7 +764,7 @@ SET (column_name1, column_name2) =(select sourceColumn1, sourceColumn2 from sour
<ul>
<li><strong>Minor Compaction</strong></li>
</ul>
-<p>In Minor compaction, user can specify the number of loads to be merged.
+<p>In Minor compaction, the user can specify the number of loads to be merged.
Minor compaction triggers for every data load if the parameter carbon.enable.auto.load.merge is set to true.
If any segments are available to be merged, then compaction will run parallel with data load, there are 2 levels in minor compaction:</p>
<ul>
@@ -746,11 +796,7 @@ Custom compaction is usually done during the off-peak time.</p>
<p>Clean the segments which are compacted:</p>
<pre><code>CLEAN FILES FOR TABLE carbon_table
</code></pre>
-
-</li>
-</ul>
-</li>
-</ul><script>
+<script>
$(function() {
// Show selected style on nav item
$('.b-nav__docs').addClass('selected');
diff --git a/content/documentation.html b/content/documentation.html
index 77505dc..da65de8 100644
--- a/content/documentation.html
+++ b/content/documentation.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -236,7 +245,7 @@
<h2>
<a id="getting-started" class="anchor" href="#getting-started" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Getting Started</h2>
<p><strong>File Format Concepts:</strong> Start with the basics of understanding the <a href="./file-structure-of-carbondata.html#carbondata-file-format">CarbonData file format</a> and its <a href="./file-structure-of-carbondata.html">storage structure</a>. This will help to understand other parts of the documentation, including deployment, programming and usage guides.</p>
-<p><strong>Quick Start:</strong> <a href="./quick-start-guide.html#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Run an example program</a> on your local machine or <a href="https://github.com/apache/carbondata/tree/master/examples/spark2/src/main/scala/org/apache/carbondata/examples" target=_blank>study some examples</a>.</p>
+<p><strong>Quick Start:</strong> <a href="./quick-start-guide.html#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Run an example program</a> on your local machine or <a href="https://github.com/apache/carbondata/tree/master/examples/spark/src/main/scala/org/apache/carbondata/examples" target=_blank>study some examples</a>.</p>
<p><strong>CarbonData SQL Language Reference:</strong> CarbonData extends the Spark SQL language and adds several <a href="./ddl-of-carbondata.html">DDL</a> and <a href="./dml-of-carbondata.html">DML</a> statements to support operations on it. Refer to the <a href="./language-manual.html">Reference Manual</a> to understand the supported features and functions.</p>
<p><strong>Programming Guides:</strong> You can read our guides about <a href="./sdk-guide.html">Java APIs supported</a> or <a href="./csdk-guide.html">C++ APIs supported</a> to learn how to integrate CarbonData with your applications.</p>
<h2>
@@ -256,7 +265,7 @@ faster data format.Contributing to CarbonData doesn?t just mean writing code. He
<p><strong>Wiki:</strong> You can read the <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Home" rel="nofollow">Apache CarbonData wiki</a> page for upcoming release plan, blogs and training materials.</p>
<p><strong>Summit:</strong> Presentations from past summits and conferences can be found <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609" rel="nofollow">here</a>.</p>
<p><strong>Blogs:</strong> Blogs by external users can be found <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635497" rel="nofollow">here</a>.</p>
-<p><strong>Performance reports:</strong> TPC-H performance reports can be found <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/TPCH+Report+of+CarbonData+%281.5.1+version%29+and+Parquet+on+Spark+Execution+Engine" rel="nofollow">here</a>.</p>
+<p><strong>Performance reports:</strong> TPC-H performance reports can be found <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Performance+-+TPCH+Report+of+CarbonData+%281.2+version%29+and+Parquet+on+Spark+Execution+Engine" rel="nofollow">here</a>.</p>
<p><strong>Trainings:</strong> Training records on design and code flows can be found <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Training+Materials" rel="nofollow">here</a>.</p>
</div>
</div>
diff --git a/content/faq.html b/content/faq.html
index 4dc17a6..c9e77f3 100644
--- a/content/faq.html
+++ b/content/faq.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -241,10 +250,10 @@
<li><a href="#what-is-carbon-lock-type">What is Carbon Lock Type?</a></li>
<li><a href="#how-to-resolve-abstract-method-error">How to resolve Abstract Method Error?</a></li>
<li><a href="#how-carbon-will-behave-when-execute-insert-operation-in-abnormal-scenarios">How Carbon will behave when execute insert operation in abnormal scenarios?</a></li>
-<li><a href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table">Why aggregate query is not fetching data from aggregate table?</a></li>
<li><a href="#why-all-executors-are-showing-success-in-spark-ui-even-after-dataload-command-failed-at-driver-side">Why all executors are showing success in Spark UI even after Dataload command failed at Driver side?</a></li>
<li><a href="#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output">Why different time zone result for select query output when query SDK writer output?</a></li>
<li><a href="#how-to-check-lru-cache-memory-footprint">How to check LRU cache memory footprint?</a></li>
+<li><a href="#How-to-deal-with-the-trailing-task-in-query">How to deal with the trailing task in query?</a></li>
</ul>
<h1>
<a id="troubleshooting" class="anchor" href="#troubleshooting" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>TroubleShooting</h1>
@@ -259,8 +268,7 @@
<li><a href="#failed-to-load-data-on-the-cluster">Failed to load data on the cluster</a></li>
<li><a href="#failed-to-insert-data-on-the-cluster">Failed to insert data on the cluster</a></li>
<li><a href="#failed-to-execute-concurrent-operations-on-table-by-multiple-workers">Failed to execute Concurrent Operations(Load,Insert,Update) on table by multiple workers</a></li>
-<li><a href="#failed-to-create-a-table-with-a-single-numeric-column">Failed to create a table with a single numeric column</a></li>
-<li><a href="#failed-to-create-datamap-and-drop-datamap-is-also-not-working">Failed to create datamap and drop datamap is also not working</a></li>
+<li><a href="#failed-to-create-index-and-drop-index-is-also-not-working">Failed to create_index and drop index is also not working</a></li>
</ul>
<h2></h2>
<h2>
@@ -353,39 +361,6 @@ id city name
<p><strong>Scenario 3</strong> :</p>
<p>When the column type in carbon table is different from the column specified in select statement. The insert operation will still success, but you may get NULL in result, because NULL will be substitute value when conversion type failed.</p>
<h2>
-<a id="why-aggregate-query-is-not-fetching-data-from-aggregate-table" class="anchor" href="#why-aggregate-query-is-not-fetching-data-from-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Why aggregate query is not fetching data from aggregate table?</h2>
-<p>Following are the aggregate queries that won't fetch data from aggregate table:</p>
-<ul>
-<li>
-<strong>Scenario 1</strong> :
-When SubQuery predicate is present in the query.</li>
-</ul>
-<p>Example:</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored as carbondata;
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
-select ctry from pop1 where ctry in (select cntry from gdp21 group by cntry);
-</code></pre>
-<ul>
-<li>
-<strong>Scenario 2</strong> :
-When aggregate function along with 'in' filter.</li>
-</ul>
-<p>Example:</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored as carbondata;
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
-select cntry, sum(gdp) from gdp21 where cntry in (select ctry from pop1) group by cntry;
-</code></pre>
-<ul>
-<li>
-<strong>Scenario 3</strong> :
-When aggregate function having 'join' with equal filter.</li>
-</ul>
-<p>Example:</p>
-<pre><code>create table gdp21(cntry smallint, gdp double, y_year date) stored as carbondata;
-create datamap ag1 on table gdp21 using 'preaggregate' as select cntry, sum(gdp) from gdp21 group by cntry;
-select cntry,sum(gdp) from gdp21,pop1 where cntry=ctry group by cntry;
-</code></pre>
-<h2>
<a id="why-all-executors-are-showing-success-in-spark-ui-even-after-dataload-command-failed-at-driver-side" class="anchor" href="#why-all-executors-are-showing-success-in-spark-ui-even-after-dataload-command-failed-at-driver-side" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Why all executors are showing success in Spark UI even after Dataload command failed at Driver side?</h2>
<p>Spark executor shows task as failed after the maximum number of retry attempts, but loading the data having bad records and BAD_RECORDS_ACTION (carbon.bad.records.action) is set as "FAIL" will attempt only once but will send the signal to driver as failed instead of throwing the exception to retry, as there is no point to retry if bad record found and BAD_RECORDS_ACTION is set to fail. Hence the Spark executor displays this one attempt as successful but the command has actually failed [...]
<h2>
@@ -411,6 +386,20 @@ TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"))
<p><strong>Note:</strong> If <code>Removed entry from InMemory LRU cache</code> are frequently observed in logs, you may have to increase the configured LRU size.</p>
<p>To observe the LRU cache from heap dump, check the heap used by CarbonLRUCache class.</p>
<h2>
+<a id="how-to-deal-with-the-trailing-task-in-query" class="anchor" href="#how-to-deal-with-the-trailing-task-in-query" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How to deal with the trailing task in query?</h2>
+<p>When tuning query performance, user may found that a few tasks slow down the overall query progress. To improve performance in such case, user can set spark.locality.wait and spark.speculation=true to enable speculation in spark, which will launch multiple task and get the result the one of the task which is finished first. Besides, user can also consider following configurations to further improve performance in this case.</p>
+<p><strong>Example:</strong></p>
+<pre><code>spark.locality.wait = 500
+spark.speculation = true
+spark.speculation.quantile = 0.75
+spark.speculation.multiplier = 5
+spark.blacklist.enabled = false
+</code></pre>
+<p><strong>Note:</strong></p>
+<p>spark.locality control data locality the value of 500 is used to shorten the waiting time of spark.</p>
+<p>spark.speculation is a group of configuration, that can monitor trailing tasks and start new tasks when conditions are met.</p>
+<p>spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.</p>
+<h2>
<a id="getting-tablestatuslock-issues-when-loading-data" class="anchor" href="#getting-tablestatuslock-issues-when-loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Getting tablestatus.lock issues When loading data</h2>
<p><strong>Symptom</strong></p>
<pre><code>17/11/11 16:48:13 ERROR LocalFileLock: main hdfs:/localhost:9000/carbon/store/default/hdfstable/tablestatus.lock (No such file or directory)
@@ -458,7 +447,7 @@ $OverrideCatalog$$overrides_$e
<p>Use the following command :</p>
</li>
</ol>
-<pre><code>mvn -Pspark-2.1 -Dspark.version {yourSparkVersion} clean package
+<pre><code>mvn -Pspark-2.4 -Dspark.version={yourSparkVersion} clean package
</code></pre>
<p>Note : Refrain from using "mvn clean package" without specifying the profile.</p>
<h2>
@@ -539,7 +528,7 @@ For example, you can use scp to copy this file to all the nodes.</p>
<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p>
<ol>
<li>
-<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster.
+<p>Copy the core-site.xml, hive-site.xml, yarn-site, carbon.properties files from the master node to all the other nodes in the cluster.
For example, you can use scp to copy this file to all the nodes.</p>
<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p>
</li>
@@ -567,7 +556,7 @@ For example, you can use scp to copy this file to all the nodes.</p>
<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p>
<ol>
<li>
-<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster.
+<p>Copy the core-site.xml, hive-site.xml, yarn-site, carbon.properties files from the master node to all the other nodes in the cluster.
For example, you can use scp to copy this file to all the nodes.</p>
<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p>
</li>
@@ -586,17 +575,7 @@ For example, you can use scp to copy this file to all the nodes.</p>
<p><strong>Procedure</strong></p>
<p>Worker must wait for the query execution to complete and the table to release the lock for another query execution to succeed.</p>
<h2>
-<a id="failed-to-create-a-table-with-a-single-numeric-column" class="anchor" href="#failed-to-create-a-table-with-a-single-numeric-column" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to create a table with a single numeric column</h2>
-<p><strong>Symptom</strong></p>
-<p>Execution fails with the following exception :</p>
-<pre><code>Table creation fails.
-</code></pre>
-<p><strong>Possible Cause</strong></p>
-<p>Behaviour not supported.</p>
-<p><strong>Procedure</strong></p>
-<p>A single column that can be considered as dimension is mandatory for table creation.</p>
-<h2>
-<a id="failed-to-create-datamap-and-drop-datamap-is-also-not-working" class="anchor" href="#failed-to-create-datamap-and-drop-datamap-is-also-not-working" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to create datamap and drop datamap is also not working</h2>
+<a id="failed-to-create-index-and-drop-index-is-also-not-working" class="anchor" href="#failed-to-create-index-and-drop-index-is-also-not-working" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to create index and drop index is also not working</h2>
<p><strong>Symptom</strong></p>
<p>Execution fails with the following exception :</p>
<pre><code>HDFS Quota Exceeded
@@ -604,8 +583,7 @@ For example, you can use scp to copy this file to all the nodes.</p>
<p><strong>Possible Cause</strong></p>
<p>HDFS Quota is set, and it is not letting carbondata write or modify any files.</p>
<p><strong>Procedure</strong></p>
-<p>Drop that particular datamap using Drop Table command using table name as
-parentTableName_datamapName so as to clear the stale folders.</p>
+<p>Drop that particular index using Drop Index command so as to clear the stale folders.</p>
<script>
// Show selected style on nav item
$(function() { $('.b-nav__faq').addClass('selected'); });
diff --git a/content/file-structure-of-carbondata.html b/content/file-structure-of-carbondata.html
index cc70c63..e2bb7a9 100644
--- a/content/file-structure-of-carbondata.html
+++ b/content/file-structure-of-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -280,7 +289,6 @@
<a id="file-content-details" class="anchor" href="#file-content-details" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>File Content details</h2>
<p>When the table is created, the user_table directory is generated, and a schema file is generated in the Metadata directory for recording the table structure.</p>
<p>When loading data in batches, each batch loading generates a new segment directory. The scheduling tries to control a task processing data loading task on each node. Each task will generate multiple carbondata files and one carbonindex file.</p>
-<p>During global dictionary generation, if the two-pass scheme is used, before the data is loaded, the corresponding dict, dictmeta and sortindex files are generated for each dictionary-encoded column, and partial dictionary files can be provided by the pre-define dictionary method to reduce the need. A dictionary-encoded column is generated by scanning the full amount of data; a dictionary file of all dictionary code columns can also be provided by the all dictionary method to avoid sc [...]
<p>The following sections use the Java object generated by the thrift file describing the carbondata file format to explain the contents of each file one by one (you can also directly read the format defined in the <a href="https://github.com/apache/carbondata/tree/master/format/src/main/thrift" target=_blank>thrift file</a>)</p>
<h3>
<a id="schema-file-format" class="anchor" href="#schema-file-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Schema file format</h3>
diff --git a/content/flink-integration-guide.html b/content/flink-integration-guide.html
new file mode 100644
index 0000000..61cf0cd
--- /dev/null
+++ b/content/flink-integration-guide.html
@@ -0,0 +1,494 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
+ target="_blank">Apache CarbonData 1.6.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.0/"
+ target="_blank">Apache CarbonData 1.6.0</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.4/"
+ target="_blank">Apache CarbonData 1.5.4</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.3/"
+ target="_blank">Apache CarbonData 1.5.3</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.2/"
+ target="_blank">Apache CarbonData 1.5.2</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"
+ target="_blank">Apache CarbonData 1.5.1</a></li>
+ <li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
+ target="_blank">Apache CarbonData 1.5.0</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html" class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
+ target="_blank">Contributing to CarbonData</a></li>
+ <li>
+ <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
+ target="_blank">Project PMC and Committers</a></li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData Security</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos </a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html"
+ target="_blank">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search" target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5" maxlength="255" value=""
+ class="search-input" placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%" colspan="2">
+ <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item" href="./csdk-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
+ <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
+ <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item" href="./security.html">Security</a>
+ <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbon-flink-integration-guide" class="anchor" href="#carbon-flink-integration-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Carbon Flink Integration Guide</h1>
+<h2>
+<a id="usage-scenarios" class="anchor" href="#usage-scenarios" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage scenarios</h2>
+<p>The CarbonData flink integration module is used to connect Flink and Carbon. The module provides
+a set of Flink BulkWriter implementations (CarbonLocalWriter and CarbonS3Writer). The data is processed
+by the Flink, and finally written into the stage directory of the target table by the CarbonXXXWriter.</p>
+<p>By default, those data in table stage directory, can not be immediately queried, those data can be queried
+after the <code>INSERT INTO $tableName STAGE</code> command is executed.</p>
+<p>Since the flink data written to carbon is endless, in order to ensure the visibility of data
+and the controllable amount of data processed during the execution of each insert form stage command,
+the user should execute the insert from stage command in a timely manner.</p>
+<p>The execution interval of the insert form stage command should take the data visibility requirements
+of the actual business and the flink data traffic. When the data visibility requirements are high
+or the data traffic is large, the execution interval should be appropriately shortened.</p>
+<p>A typical scenario is that the data is cleaned and preprocessed by Flink, and then written to Carbon,
+for subsequent analysis and queries.</p>
+<h2>
+<a id="usage-description" class="anchor" href="#usage-description" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage description</h2>
+<h3>
+<a id="writing-process" class="anchor" href="#writing-process" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writing process</h3>
+<p>Typical flink stream: <code>Source -> Process -> Output(Carbon Writer Sink)</code></p>
+<p>Pseudo code and description:</p>
+<div class="highlight highlight-source-scala"><pre> <span class="pl-c"><span class="pl-c">//</span> Import dependencies.</span>
+ <span class="pl-k">import</span> <span class="pl-en">java</span>.<span class="pl-en">util</span>.<span class="pl-en">Properties</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbon</span>.<span class="pl-en">flink</span>.<span class="pl-en">CarbonWriterFactory</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbon</span>.<span class="pl-en">flink</span>.<span class="pl-en">ProxyFileSystem</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbondata</span>.<span class="pl-en">core</span>.<span class="pl-en">constants</span>.<span class="pl-en">CarbonCommonConstants</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">api</span>.<span class="pl-en">common</span>.<span class="pl-en">restartstrategy</span>.<span class="pl-en">RestartStrategies</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">core</span>.<span class="pl-en">fs</span>.<span class="pl-en">Path</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">streaming</span>.<span class="pl-en">api</span>.<span class="pl-en">environment</span>.<span class="pl-en">StreamExecutionEnvironment</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">streaming</span>.<span class="pl-en">api</span>.<span class="pl-en">functions</span>.<span class="pl-en">sink</span>.<span class="pl-en">filesystem</span>.<span class="pl-en">StreamingFileSink</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Specify database name.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">databaseName</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>default<span class="pl-pds">"</span></span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Specify target table name.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">tableName</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
+ <span class="pl-c"><span class="pl-c">//</span> Table path of the target table.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">tablePath</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/data/warehouse/test<span class="pl-pds">"</span></span>
+ <span class="pl-c"><span class="pl-c">//</span> Specify local temporary path.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">dataTempPath</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/data/temp/<span class="pl-pds">"</span></span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">tableProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+ <span class="pl-c"><span class="pl-c">//</span> Set the table properties here.</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">writerProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+ <span class="pl-c"><span class="pl-c">//</span> Set the writer properties here, such as temp path, commit threshold, access key, secret key, endpoint, etc.</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">carbonProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+ <span class="pl-c"><span class="pl-c">//</span> Set the carbon properties here, such as date format, store location, etc.</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">writerFactory</span> <span class="pl-k">=</span> <span class="pl-en">CarbonWriterFactory</span>.builder(<span class="pl-s"><span class="pl-pds">"</span>Local<span class="pl-pds">"</span></span>).build(
+ databaseName,
+ tableName,
+ tablePath,
+ tableProperties,
+ writerProperties,
+ carbonProperties
+ )
+
+ <span class="pl-c"><span class="pl-c">//</span> Build a flink stream and run it.</span>
+ <span class="pl-c"><span class="pl-c">//</span> 1. Create a new flink execution environment.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">environment</span> <span class="pl-k">=</span> <span class="pl-en">StreamExecutionEnvironment</span>.getExecutionEnvironment
+ <span class="pl-c"><span class="pl-c">//</span> Set flink environment configuration here, such as parallelism, checkpointing, restart strategy, etc.</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> 2. Create flink data source, may be a kafka source, custom source, or others.</span>
+ <span class="pl-c"><span class="pl-c">//</span> The data type of source should be Array[AnyRef].</span>
+ <span class="pl-c"><span class="pl-c">//</span> Array length should equals to table column count, and values order in array should matches table column order.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">source</span> <span class="pl-k">=</span> ...
+ <span class="pl-c"><span class="pl-c">//</span> 3. Create flink stream and set source.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">stream</span> <span class="pl-k">=</span> environment.addSource(source)
+ <span class="pl-c"><span class="pl-c">//</span> 4. Add other flink operators here.</span>
+ <span class="pl-c"><span class="pl-c">//</span> ...</span>
+ <span class="pl-c"><span class="pl-c">//</span> 5. Set flink stream target (write data to carbon with a write sink).</span>
+ stream.addSink(<span class="pl-en">StreamingFileSink</span>.forBulkFormat(<span class="pl-k">new</span> <span class="pl-en">Path</span>(<span class="pl-en">ProxyFileSystem</span>.<span class="pl-en">DEFAULT_URI</span>), writerFactory).build)
+ <span class="pl-c"><span class="pl-c">//</span> 6. Run flink stream.</span>
+ <span class="pl-k">try</span> {
+ environment.execute
+ } <span class="pl-k">catch</span> {
+ <span class="pl-k">case</span> <span class="pl-v">exception</span>: <span class="pl-en">Exception</span> <span class="pl-k">=></span>
+ <span class="pl-c"><span class="pl-c">//</span> Handle execute exception here.</span>
+ }</pre></div>
+<h3>
+<a id="writer-properties" class="anchor" href="#writer-properties" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writer properties</h3>
+<h4>
+<a id="local-writer" class="anchor" href="#local-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Local Writer</h4>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>CarbonLocalProperty.DATA_TEMP_PATH</td>
+<td>carbon.writer.local.data.temp.path</td>
+<td>Usually is a local path, data will write to temp path first, and move to target data path finally.</td>
+</tr>
+<tr>
+<td>CarbonLocalProperty.COMMIT_THRESHOLD</td>
+<td>carbon.writer.local.commit.threshold</td>
+<td>While written data count reach the threshold, data writer will flush and move data to target data path.</td>
+</tr>
+</tbody>
+</table>
+<h4>
+<a id="s3-writer" class="anchor" href="#s3-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Writer</h4>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>CarbonS3Property.ACCESS_KEY</td>
+<td>carbon.writer.s3.access.key</td>
+<td>Access key of s3 file system</td>
+</tr>
+<tr>
+<td>CarbonS3Property.SECRET_KEY</td>
+<td>carbon.writer.s3.secret.key</td>
+<td>Secret key of s3 file system</td>
+</tr>
+<tr>
+<td>CarbonS3Property.ENDPOINT</td>
+<td>carbon.writer.s3.endpoint</td>
+<td>Endpoint of s3 file system</td>
+</tr>
+<tr>
+<td>CarbonS3Property.DATA_TEMP_PATH</td>
+<td>carbon.writer.s3.data.temp.path</td>
+<td>Usually is a local path, data will write to temp path first, and move to target data path finally.</td>
+</tr>
+<tr>
+<td>CarbonS3Property.COMMIT_THRESHOLD</td>
+<td>carbon.writer.s3.commit.threshold</td>
+<td>While written data count reach the threshold, data writer will flush and move data to target data path.</td>
+</tr>
+</tbody>
+</table>
+<h3>
+<a id="insert-from-stage" class="anchor" href="#insert-from-stage" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Insert from stage</h3>
+<p>Refer <a href="./dml-of-carbondata.html#insert-data-into-carbondata-table-from-stage-input-files">Grammar Description</a> for syntax.</p>
+<h2>
+<a id="usage-example-code" class="anchor" href="#usage-example-code" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Usage Example Code</h2>
+<p>Create target table.</p>
+<div class="highlight highlight-source-sql"><pre> <span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">test</span> (col1 string, col2 string, col3 string) STORED <span class="pl-k">AS</span> carbondata</pre></div>
+<p>Writing flink data to local carbon table.</p>
+<div class="highlight highlight-source-scala"><pre> <span class="pl-k">import</span> <span class="pl-en">java</span>.<span class="pl-en">util</span>.<span class="pl-en">Properties</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbon</span>.<span class="pl-en">flink</span>.<span class="pl-en">CarbonLocalProperty</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbon</span>.<span class="pl-en">flink</span>.<span class="pl-en">CarbonWriterFactory</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbon</span>.<span class="pl-en">flink</span>.<span class="pl-en">ProxyFileSystem</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbondata</span>.<span class="pl-en">core</span>.<span class="pl-en">constants</span>.<span class="pl-en">CarbonCommonConstants</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">api</span>.<span class="pl-en">common</span>.<span class="pl-en">restartstrategy</span>.<span class="pl-en">RestartStrategies</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">core</span>.<span class="pl-en">fs</span>.<span class="pl-en">Path</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">streaming</span>.<span class="pl-en">api</span>.<span class="pl-en">environment</span>.<span class="pl-en">StreamExecutionEnvironment</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">streaming</span>.<span class="pl-en">api</span>.<span class="pl-en">functions</span>.<span class="pl-en">sink</span>.<span class="pl-en">filesystem</span>.<span class="pl-en">StreamingFileSink</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">flink</span>.<span class="pl-en">streaming</span>.<span class="pl-en">api</span>.<span class="pl-en">functions</span>.<span class="pl-en">source</span>.<span class="pl-en">SourceFunction</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">databaseName</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>default<span class="pl-pds">"</span></span>
+ <span class="pl-k">val</span> <span class="pl-smi">tableName</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
+ <span class="pl-k">val</span> <span class="pl-smi">tablePath</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/data/warehouse/test<span class="pl-pds">"</span></span>
+ <span class="pl-k">val</span> <span class="pl-smi">dataTempPath</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/data/temp/<span class="pl-pds">"</span></span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">tableProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">writerProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+ writerProperties.setProperty(<span class="pl-en">CarbonLocalProperty</span>.<span class="pl-en">DATA_TEMP_PATH</span>, dataTempPath)
+
+ <span class="pl-k">val</span> <span class="pl-smi">carbonProperties</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Properties</span>
+ carbonProperties.setProperty(<span class="pl-en">CarbonCommonConstants</span>.<span class="pl-en">CARBON_TIMESTAMP_FORMAT</span>, <span class="pl-en">CarbonCommonConstants</span>.<span class="pl-en">CARBON_TIMESTAMP_DEFAULT_FORMAT</span>)
+ carbonProperties.setProperty(<span class="pl-en">CarbonCommonConstants</span>.<span class="pl-en">CARBON_DATE_FORMAT</span>, <span class="pl-en">CarbonCommonConstants</span>.<span class="pl-en">CARBON_DATE_DEFAULT_FORMAT</span>)
+ carbonProperties.setProperty(<span class="pl-en">CarbonCommonConstants</span>.<span class="pl-en">UNSAFE_WORKING_MEMORY_IN_MB</span>, <span class="pl-s"><span class="pl-pds">"</span>1024<span class="pl-pds">"</span></span>)
+
+ <span class="pl-k">val</span> <span class="pl-smi">writerFactory</span> <span class="pl-k">=</span> <span class="pl-en">CarbonWriterFactory</span>.builder(<span class="pl-s"><span class="pl-pds">"</span>Local<span class="pl-pds">"</span></span>).build(
+ databaseName,
+ tableName,
+ tablePath,
+ tableProperties,
+ writerProperties,
+ carbonProperties
+ )
+
+ <span class="pl-k">val</span> <span class="pl-smi">environment</span> <span class="pl-k">=</span> <span class="pl-en">StreamExecutionEnvironment</span>.getExecutionEnvironment
+ environment.setParallelism(<span class="pl-c1">1</span>)
+ environment.enableCheckpointing(<span class="pl-c1">2000L</span>)
+ environment.setRestartStrategy(<span class="pl-en">RestartStrategies</span>.noRestart)
+
+ <span class="pl-c"><span class="pl-c">//</span> Define a custom source.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">source</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">SourceFunction</span>[<span class="pl-en">Array</span>[<span class="pl-en">AnyRef</span>]]() {
+ <span class="pl-k">override</span>
+ <span class="pl-k">def</span> <span class="pl-en">run</span>(<span class="pl-v">sourceContext</span>: <span class="pl-en">SourceFunction</span>.<span class="pl-en">SourceContext</span>[<span class="pl-en">Array</span>[<span class="pl-en">AnyRef</span>]])<span class="pl-k">:</span> <span class="pl-en">Unit</span> <span class="pl-k">=</span> {
+ <span class="pl-c"><span class="pl-c">//</span> Array length should equals to table column count, and values order in array should matches table column order.</span>
+ <span class="pl-k">val</span> <span class="pl-smi">data</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Array</span>[<span class="pl-en">AnyRef</span>](<span class="pl-c1">3</span>)
+ data(<span class="pl-c1">0</span>) <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>value1<span class="pl-pds">"</span></span>
+ data(<span class="pl-c1">1</span>) <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>value2<span class="pl-pds">"</span></span>
+ data(<span class="pl-c1">2</span>) <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>value3<span class="pl-pds">"</span></span>
+ sourceContext.collect(data)
+ }
+
+ <span class="pl-k">override</span>
+ <span class="pl-k">def</span> <span class="pl-en">cancel</span>()<span class="pl-k">:</span> <span class="pl-en">Unit</span> <span class="pl-k">=</span> {
+ <span class="pl-c"><span class="pl-c">//</span> do something.</span>
+ }
+ }
+
+ <span class="pl-k">val</span> <span class="pl-smi">stream</span> <span class="pl-k">=</span> environment.addSource(source)
+ <span class="pl-k">val</span> <span class="pl-smi">streamSink</span> <span class="pl-k">=</span> <span class="pl-en">StreamingFileSink</span>.forBulkFormat(<span class="pl-k">new</span> <span class="pl-en">Path</span>(<span class="pl-en">ProxyFileSystem</span>.<span class="pl-en">DEFAULT_URI</span>), writerFactory).build
+
+ stream.addSink(streamSink)
+
+ <span class="pl-k">try</span> {
+ environment.execute
+ } <span class="pl-k">catch</span> {
+ <span class="pl-k">case</span> <span class="pl-v">exception</span>: <span class="pl-en">Exception</span> <span class="pl-k">=></span>
+ <span class="pl-c"><span class="pl-c">//</span> TODO</span>
+ <span class="pl-k">throw</span> <span class="pl-k">new</span> <span class="pl-en">UnsupportedOperationException</span>(exception)
+ }</pre></div>
+<p>Insert into table from stage directory.</p>
+<div class="highlight highlight-source-sql"><pre> <span class="pl-k">INSERT INTO</span> test STAGE</pre></div>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__docs').addClass('selected');
+
+ // Display docs subnav items
+ if (!$('.b-nav__docs').parent().hasClass('nav__item__with__subs--expanded')) {
+ $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/hive-guide.html b/content/hive-guide.html
index 9fc7a51..1f3ad4b 100644
--- a/content/hive-guide.html
+++ b/content/hive-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -273,12 +282,12 @@ val metaStoreDB = s"$rootPath/metastore_db"
val carbon = SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, storeLocation).getOrCreateCarbonSession(storeLocation, metaStoreDB)
-carbon.sql("create table hive_carbon(id int, name string, scale decimal, country string, salary double) STORED BY 'carbondata'")
+carbon.sql("create table hive_carbon(id int, name string, scale decimal, country string, salary double) STORED AS carbondata")
carbon.sql("LOAD DATA INPATH '<hdfs store path>/sample.csv' INTO TABLE hive_carbon")
scala>carbon.sql("SELECT * FROM hive_carbon").show()
</code></pre>
<h2>
-<a id="query-data-in-hive" class="anchor" href="#query-data-in-hive" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data in Hive</h2>
+<a id="configure-carbon-in-hive" class="anchor" href="#configure-carbon-in-hive" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configure Carbon in Hive</h2>
<h3>
<a id="configure-hive-classpath" class="anchor" href="#configure-hive-classpath" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configure hive classpath</h3>
<pre><code>mkdir hive/auxlibs/
@@ -305,6 +314,15 @@ Carbon Jars to be copied to the above paths.
<pre><code>$HIVE_HOME/bin/beeline
</code></pre>
<h3>
+<a id="write-data-from-hive" class="anchor" href="#write-data-from-hive" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Write data from hive</h3>
+<ul>
+<li>Write data from hive into carbondata format.</li>
+</ul>
+<pre><code>create table hive_carbon(id int, name string, scale decimal, country string, salary double) stored by 'org.apache.carbondata.hive.CarbonStorageHandler';
+insert into hive_carbon select * from parquetTable;
+</code></pre>
+<p><strong>Note</strong>: Only non-transactional tables are supported when created through hive. This means that the standard carbon folder structure would not be followed and all files would be written in a flat folder structure.</p>
+<h3>
<a id="query-data-from-hive" class="anchor" href="#query-data-from-hive" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query data from hive</h3>
<ul>
<li>This is to read the carbon table through Hive. It is the integration of the carbon with Hive.</li>
@@ -315,19 +333,15 @@ These properties helps to recursively traverse through the directories to read t
</code></pre>
<h3>
<a id="example" class="anchor" href="#example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example</h3>
-<pre><code> - In case if the carbon table is not set with the SERDE and the INPUTFORMAT/OUTPUTFORMAT, user can create a new hive managed table like below with the required details for the hive to read.
-create table hive_carbon_1(id int, name string, scale decimal, country string, salary double) ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', 'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE') STORED AS INPUTFORMAT 'org.apache.carbondata.hive.MapredCarbonInputFormat' OUTPUTFORMAT 'org.apache.carbondata.hive.MapredCarbonOutputFormat' LOCATION 'location_to_the_carbon_table';
-
- - Query the table
-select * from hive_carbon_1;
-select count(*) from hive_carbon_1;
-select * from hive_carbon_1 order by id;
+<pre><code> - Query the table
+select * from hive_carbon;
+select count(*) from hive_carbon;
+select * from hive_carbon order by id;
</code></pre>
<h3>
<a id="note" class="anchor" href="#note" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Note</h3>
<ul>
<li>Partition table support is not handled</li>
-<li>Map data type is not supported</li>
</ul>
<script>
// Show selected style on nav item
diff --git a/content/how-to-contribute-to-apache-carbondata.html b/content/how-to-contribute-to-apache-carbondata.html
index b1fa046..f27f17b 100644
--- a/content/how-to-contribute-to-apache-carbondata.html
+++ b/content/how-to-contribute-to-apache-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
diff --git a/content/datamap-developer-guide.html b/content/index-developer-guide.html
similarity index 84%
rename from content/datamap-developer-guide.html
rename to content/index-developer-guide.html
index aa678aa..b43f156 100644
--- a/content/datamap-developer-guide.html
+++ b/content/index-developer-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,26 +240,25 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="datamap-developer-guide" class="anchor" href="#datamap-developer-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Developer Guide</h1>
+<a id="index-developer-guide" class="anchor" href="#index-developer-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Developer Guide</h1>
<h3>
<a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Introduction</h3>
-<p>DataMap is a data structure that can be used to accelerate certain query of the table. Different DataMap can be implemented by developers.
-Currently, there are two types of DataMap supported:</p>
+<p>Index is a data structure that can be used to accelerate certain query of the table. Different Index can be implemented by developers.
+Currently, Carbondata supports three types of Indexes:</p>
<ol>
-<li>IndexDataMap: DataMap that leverages index to accelerate filter query. Lucene DataMap and BloomFiler DataMap belong to this type of DataMaps.</li>
-<li>MVDataMap: DataMap that leverages Materialized View to accelerate olap style query, like SPJG query (select, predicate, join, groupby). Preaggregate, timeseries and mv DataMap belong to this type of DataMaps.</li>
+<li>BloomFilter Index: A space-efficient probabilistic data structure that is used to test whether an element is a member of a set.</li>
+<li>Lucene Index: High performance, full-featured text search engine.</li>
+<li>Secondary Index: Sencondary index tables to hold blocklets are created as indexes and managed as child tables internally by Carbondata.</li>
</ol>
<h3>
-<a id="datamap-provider" class="anchor" href="#datamap-provider" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Provider</h3>
-<p>When user issues <code>CREATE DATAMAP dm ON TABLE main USING 'provider'</code>, the corresponding DataMapProvider implementation will be created and initialized.
+<a id="index-provider" class="anchor" href="#index-provider" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Provider</h3>
+<p>When user issues <code>CREATE INDEX index_name ON TABLE main AS 'provider'</code>, the corresponding IndexProvider implementation will be created and initialized.
Currently, the provider string can be:</p>
<ol>
-<li>preaggregate: A type of MVDataMap that do pre-aggregate of single table</li>
-<li>timeseries: A type of MVDataMap that do pre-aggregate based on time dimension of the table</li>
-<li>class name IndexDataMapFactory implementation: Developer can implement new type of IndexDataMap by extending IndexDataMapFactory</li>
+<li>class name IndexFactory implementation: Developer can implement new type of Index by extending IndexFactory</li>
</ol>
-<p>When user issues <code>DROP DATAMAP dm ON TABLE main</code>, the corresponding DataMapProvider interface will be called.</p>
-<p>Click for more details about <a href="./datamap-management.html#datamap-management">DataMap Management</a> and supported <a href="./datamap-management.html#overview">DSL</a>.</p>
+<p>When user issues <code>DROP INDEX index_name ON TABLE main</code>, the corresponding IndexFactory class will be called.</p>
+<p>Click for more details about <a href="./index/index-management.html#index-management">Index Management</a> and supported <a href="./index/index-management.html#overview">DSL</a>.</p>
<script>
$(function() {
// Show selected style on nav item
diff --git a/src/main/webapp/datamap-management.html b/content/index-management.html
similarity index 70%
rename from src/main/webapp/datamap-management.html
rename to content/index-management.html
index db96959..0794aa7 100644
--- a/src/main/webapp/datamap-management.html
+++ b/content/index-management.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,155 +240,121 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-datamap-management" class="anchor" href="#carbondata-datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData DataMap Management</h1>
+<a id="carbondata-index-management" class="anchor" href="#carbondata-index-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Index Management</h1>
<ul>
<li><a href="#overview">Overview</a></li>
-<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#index-management">Index Management</a></li>
<li><a href="#automatic-refresh">Automatic Refresh</a></li>
<li><a href="#manual-refresh">Manual Refresh</a></li>
-<li><a href="#datamap-catalog">DataMap Catalog</a></li>
<li>
-<a href="#datamap-related-commands">DataMap Related Commands</a>
+<a href="#index-related-commands">Index Related Commands</a>
<ul>
<li><a href="#explain">Explain</a></li>
-<li><a href="#show-datamap">Show DataMap</a></li>
-<li><a href="#compaction-on-datamap">Compaction on DataMap</a></li>
+<li><a href="#show-index">Show Index</a></li>
</ul>
</li>
</ul>
<h2>
<a id="overview" class="anchor" href="#overview" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Overview</h2>
-<p>DataMap can be created using following DDL</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-[ON TABLE main_table]
-USING "datamap_provider"
-[WITH DEFERRED REBUILD]
-DMPROPERTIES ('key'='value', ...)
-AS
- SELECT statement
+<p>Index can be created using following DDL</p>
+<pre><code>CREATE INDEX [IF NOT EXISTS] index_name
+ON TABLE [db_name.]table_name (column_name, ...)
+AS carbondata/bloomfilter/lucene
+[WITH DEFERRED REFRESH]
+[PROPERTIES ('key'='value')]
+</code></pre>
+<p>Index can be refreshed using following DDL</p>
+<pre><code>REFRESH INDEX index_name ON [TABLE] [db_name.]table_name
</code></pre>
-<p>Currently, there are 5 DataMap implementations in CarbonData.</p>
+<p>Currently, there are 3 Index implementations in CarbonData.</p>
<table>
<thead>
<tr>
-<th>DataMap Provider</th>
+<th>Index Provider</th>
<th>Description</th>
-<th>DMPROPERTIES</th>
<th>Management</th>
</tr>
</thead>
<tbody>
<tr>
-<td>preaggregate</td>
-<td>single table pre-aggregate table</td>
-<td>No DMPROPERTY is required</td>
-<td>Automatic</td>
-</tr>
-<tr>
-<td>timeseries</td>
-<td>time dimension rollup table</td>
-<td>event_time, xx_granularity, please refer to <a href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
-</td>
+<td>secondary-index</td>
+<td>secondary-index tables to hold blocklets as indexes and managed as child tables</td>
<td>Automatic</td>
</tr>
<tr>
-<td>mv</td>
-<td>multi-table pre-aggregate table</td>
-<td>No DMPROPERTY is required</td>
-<td>Manual/Automatic</td>
-</tr>
-<tr>
<td>lucene</td>
<td>lucene indexing for text column</td>
-<td>index_columns to specifying the index columns</td>
<td>Automatic</td>
</tr>
<tr>
<td>bloomfilter</td>
<td>bloom filter for high cardinality column, geospatial column</td>
-<td>index_columns to specifying the index columns</td>
<td>Automatic</td>
</tr>
</tbody>
</table>
<h2>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h2>
-<p>There are two kinds of management semantic for DataMap.</p>
+<a id="index-management" class="anchor" href="#index-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Management</h2>
+<p>There are two kinds of management semantic for Index.</p>
<ol>
-<li>Automatic Refresh: Create datamap without <code>WITH DEFERRED REBUILD</code> in the statement, which is by default.</li>
-<li>Manual Refresh: Create datamap with <code>WITH DEFERRED REBUILD</code> in the statement</li>
+<li>Automatic Refresh</li>
+<li>Manual Refresh</li>
</ol>
<h3>
<a id="automatic-refresh" class="anchor" href="#automatic-refresh" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Automatic Refresh</h3>
-<p>When user creates a datamap on the main table without using <code>WITH DEFERRED REBUILD</code> syntax, the datamap will be managed by system automatically.
-For every data load to the main table, system will immediately trigger a load to the datamap automatically. These two data loading (to main table and datamap) is executed in a transactional manner, meaning that it will be either both success or neither success.</p>
-<p>The data loading to datamap is incremental based on Segment concept, avoiding a expensive total rebuild.</p>
-<p>If user perform following command on the main table, system will return failure. (reject the operation)</p>
+<p>When a user creates an index on the main table without using <code>WITH DEFERRED REFRESH</code> syntax, the index will be managed by the system automatically.
+For every data load to the main table, the system will immediately trigger a load to the index automatically. These two data loading (to main table and index) is executed in a transactional manner, meaning that it will be either both success or neither success.</p>
+<p>The data loading to index is incremental based on Segment concept, avoiding an expensive total refresh.</p>
+<p>If a user performs the following command on the main table, the system will return failure. (reject the operation)</p>
<ol>
<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
-change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
-not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
+change datatype command, CarbonData will check whether it will impact the index table, if
+not, the operation is allowed, otherwise operation will be rejected by throwing an exception.</li>
<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>.</li>
</ol>
-<p>If user do want to perform above operations on the main table, user can first drop the datamap, perform the operation, and re-create the datamap again.</p>
-<p>If user drop the main table, the datamap will be dropped immediately too.</p>
-<p>We do recommend you to use this management for index datamap.</p>
+<p>If a user does want to perform above operations on the main table, the user can first drop the index, perform the operation, and re-create the index again.</p>
+<p>If a user drops the main table, the index will be dropped immediately too.</p>
+<p>We do recommend you to use this management for indexing.</p>
<h3>
<a id="manual-refresh" class="anchor" href="#manual-refresh" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Manual Refresh</h3>
-<p>When user creates a datamap specifying manual refresh semantic, the datamap is created with status <em>disabled</em> and query will NOT use this datamap until user can issue REBUILD DATAMAP command to build the datamap. For every REBUILD DATAMAP command, system will trigger a full rebuild of the datamap. After rebuild is done, system will change datamap status to <em>enabled</em>, so that it can be used in query rewrite.</p>
-<p>For every new data loading, data update, delete, the related datamap will be made <em>disabled</em>,
-which means that the following queries will not benefit from the datamap before it becomes <em>enabled</em> again.</p>
-<p>If the main table is dropped by user, the related datamap will be dropped immediately.</p>
+<p>When a user creates an index on the main table using <code>WITH DEFERRED REFRESH</code> syntax, the index will be created with status <em>disabled</em> and query will NOT use this index until the user issues <code>REFRESH INDEX</code> command to build the index. For every <code>REFRESH INDEX</code> command, the system will trigger a full refresh of the index. Once the refresh operation is finished, system will change index status to <em>enabled</em>, so that it can be used in query re [...]
+<p>For every new data loading, data update, delete, the related index will be made <em>disabled</em>,
+which means that the following queries will not benefit from the index before it becomes <em>enabled</em> again.</p>
+<p>If the main table is dropped by the user, the related index will be dropped immediately.</p>
<p><strong>Note</strong>:</p>
<ul>
-<li>If you are creating a datamap on external table, you need to do manual management of the datamap.</li>
-<li>For index datamap such as BloomFilter datamap, there is no need to do manual refresh.
-By default it is automatic refresh,
-which means its data will get refreshed immediately after the datamap is created or the main table is loaded.
-Manual refresh on this datamap will has no impact.</li>
-</ul>
-<h2>
-<a id="datamap-catalog" class="anchor" href="#datamap-catalog" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Catalog</h2>
-<p>Currently, when user creates a datamap, system will store the datamap metadata in a configurable <em>system</em> folder in HDFS or S3.</p>
-<p>In this <em>system</em> folder, it contains:</p>
-<ul>
-<li>DataMapSchema file. It is a json file containing schema for one datamap. Ses DataMapSchema class. If user creates 100 datamaps (on different tables), there will be 100 files in <em>system</em> folder.</li>
-<li>DataMapStatus file. Only one file, it is in json format, and each entry in the file represents for one datamap. Ses DataMapStatusDetail class</li>
+<li>If you are creating an index on an external table, you need to do manual management of the index.</li>
+<li>Currently, all types of indexes supported by carbon will be automatically refreshed by default, which means its data will get refreshed immediately after the index is created or the main table is loaded. Manual refresh on these indexes is not supported.</li>
</ul>
-<p>There is a DataMapCatalog interface to retrieve schema of all datamap, it can be used in optimizer to get the metadata of datamap.</p>
<h2>
-<a id="datamap-related-commands" class="anchor" href="#datamap-related-commands" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Related Commands</h2>
+<a id="index-related-commands" class="anchor" href="#index-related-commands" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Related Commands</h2>
<h3>
<a id="explain" class="anchor" href="#explain" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Explain</h3>
-<p>How can user know whether datamap is used in the query?</p>
-<p>User can set enable.query.statistics = true and use EXPLAIN command to know, it will print out something like</p>
+<p>How can users know whether an index is used in the query?</p>
+<p>User can set <code>enable.query.statistics = true</code> and use <code>EXPLAIN</code> command to know, it will print out something like</p>
<pre lang="text"><code>== CarbonData Profiler ==
-Hit mv DataMap: datamap1
-Scan Table: default.datamap1_table
+Table Scan on default.main
++- total: 1 blocks, 1 blocklets
+- filter:
-+- pruning by CG DataMap
-+- all blocklets: 1
- skipped blocklets: 0
++- pruned by CG Index
+ - name: index1
+ - provider: lucene
+ - skipped: 0 blocks, 0 blocklets
</code></pre>
<h3>
-<a id="show-datamap" class="anchor" href="#show-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Show DataMap</h3>
-<p>There is a SHOW DATAMAPS command, when this is issued, system will read all datamap from <em>system</em> folder and print all information on screen. The current information includes:</p>
+<a id="show-index" class="anchor" href="#show-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Show Index</h3>
+<p>There is a SHOW INDEXES command, when this is issued, the system will read all indexes from the carbon table and print all information on screen. The current information includes:</p>
<ul>
-<li>DataMapName</li>
-<li>DataMapProviderName like mv, preaggreagte, timeseries, etc</li>
-<li>Associated Table</li>
-<li>DataMap Properties</li>
-<li>DataMap status (ENABLED/DISABLED)</li>
-<li>Sync Status - which displays Last segment Id of main table synced with datamap table and its load
-end time (Applicable only for mv datamap)</li>
+<li>Name</li>
+<li>Provider like lucene</li>
+<li>Indexed Columns</li>
+<li>Properties</li>
+<li>Status (ENABLED/DISABLED)</li>
+<li>Sync Info - which displays Last segment Id of main table synced with index table and its load
+end time</li>
</ul>
-<h3>
-<a id="compaction-on-datamap" class="anchor" href="#compaction-on-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compaction on DataMap</h3>
-<p>This feature applies for preaggregate datamap only</p>
-<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong> compact the pre-aggregate tables created on the main table. User need to run Compaction command separately on each pre-aggregate table to compact them.</p>
-<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on main table but not performed on pre-aggregate table, all queries still can benefit from pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables can be triggered to merge the segments and files in the pre-aggregate tables.</p>
<script>
$(function() {
// Show selected style on nav item
diff --git a/content/index-server.html b/content/index-server.html
index 6916187..632680d 100644
--- a/content/index-server.html
+++ b/content/index-server.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -234,9 +243,8 @@
<a id="distributed-index-server" class="anchor" href="#distributed-index-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Distributed Index Server</h1>
<h2>
<a id="background" class="anchor" href="#background" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Background</h2>
-<p>Carbon currently prunes and caches all block/blocklet datamap index information into the driver for
-normal table, for Bloom/Index datamaps the JDBC driver will launch a job to prune and cache the
-datamaps in executors.</p>
+<p>Carbon currently prunes and caches all block/blocklet index information into the driver for
+normal table, for Bloom/Lucene indexes the JDBC driver will launch a job to prune and cache in executors.</p>
<p>This causes the driver to become a bottleneck in the following ways:</p>
<ol>
<li>If the cache size becomes huge(70-80% of the driver memory) then there can be excessive GC in
@@ -268,8 +276,7 @@ cache location for each segment.</p>
</li>
<li>
<p>Cache size held by each executor:
-This mapping will be used to distribute the segments equally(on the basis of size) among the
-executors.</p>
+This mapping will be used to distribute the segments equally(on the basis of size) among the executors.</p>
</li>
</ol>
<p>Once a request is received each segment would be iterated over and
@@ -294,6 +301,14 @@ each executor which would be updated to the executorToCacheMapping and
the pruned blocklets which would be further used for result fetching.</p>
<p><strong>Note:</strong> Multiple JDBC drivers can connect to the index server to use the cache.</p>
<h2>
+<a id="enabling-size-based-distribution-for-legacy-stores" class="anchor" href="#enabling-size-based-distribution-for-legacy-stores" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Enabling Size based distribution for Legacy stores</h2>
+<p>The default round robin based distribution causes unequal distribution of cache among the executors, which can cause any one of the executors to be bloated with too much cache resulting in performance degrade.
+This problem can be solved by running the <code>upgrade_segment</code> command which will fill the data size values for each segment in the tablestatus file. Any cache loaded after this can use the traditional size based distribution.</p>
+<h4>
+<a id="example" class="anchor" href="#example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example</h4>
+<pre><code>alter table table1 compact 'upgrade_segment';
+</code></pre>
+<h2>
<a id="reallocation-of-executor" class="anchor" href="#reallocation-of-executor" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Reallocation of executor</h2>
<p>In case executor(s) become dead/unavailable then the segments that were
earlier being handled by those would be reassigned to some other
@@ -312,7 +327,7 @@ dynamic configuration to clear the cache of the desired location.</p>
which means that the JDBCServer would take care of distributed pruning.
A similar job would be fired by the JDBCServer which would take care of
pruning using its own executors. If for any reason the embedded mode
-also fails to prune the datamaps then the job would be passed on to
+also fails to prune the indexes then the job would be passed on to
driver.</p>
<p><strong>NOTE:</strong> In case of embedded mode a job would be fired after pruning to clear the
cache as data cached in JDBCServer executors would be of no use.</p>
@@ -324,17 +339,18 @@ read this file and create the splits. This can be controlled using the property
meaning that no matter how small the splits are they would be written to the file. Maximum is
102400KB which will mean if the size of the splits for a executor cross this value then they would
be written to file.</p>
-<p>The user can set the location for these file by using 'carbon.indexserver.temp.path'. By default
-table path would be used to write the files.</p>
+<p>The user can set the location for these files by using 'carbon.indexserver.temp.path'. By default
+the files are written in the path /tmp/indexservertmp.</p>
<h2>
-<a id="security" class="anchor" href="#security" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Security</h2>
-<p>The security for the index server is controlled through 'spark.carbon.indexserver.keytab' and 'spark
-.carbon.indexserver.principal'. These allow the RPC framework to login using the principal. It is
-recommended that the principal should be a super user, and the user should be exclusive for index
-server so that it does not grant access to any other service. Internally the operations would be
-executed as a Privileged Action using the login user.</p>
-<p>The Index Server is a long running service therefore the 'spark.yarn.keytab' and 'spark.yarn
-.principal' should be configured.</p>
+<a id="prepriming" class="anchor" href="#prepriming" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prepriming</h2>
+<p>As each query is responsible for caching the pruned indexes, thus a lot of execution time is wasted in reading the
+files and caching the datmaps for the first query.
+To avoid this problem we have introduced Pre-Priming which allows each data manipulation command like load, insert etc
+to fire a request to the index server to load the corresponding segments into the index server.
+When index server receives a request it checks whether the request is for pre-priming, if it is then the request is
+processed in a new thread, and a dummy response is immediately returned to the client.
+Since pre-priming acts as an async call, it does not have any negative performance impacts.</p>
+<p>The user can enable prepriming by using 'carbon.indexserver.enable.prepriming' = 'true/false'. By default this is set as false.</p>
<h2>
<a id="configurations" class="anchor" href="#configurations" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configurations</h2>
<h5>
@@ -374,6 +390,11 @@ executed as a Privileged Action using the login user.</p>
<td align="center">NA</td>
<td align="center">The max length of the job to show in the index server service UI. For bigger queries this may impact performance as the whole string would be sent from JDBCServer to IndexServer.</td>
</tr>
+<tr>
+<td align="center">carbon.indexserver.enable.prepriming</td>
+<td align="center">false</td>
+<td align="center">Enable the use of prepriming in the Index Server to improve the performance of first time query.</td>
+</tr>
</tbody>
</table>
<h5>
@@ -406,11 +427,11 @@ executed as a Privileged Action using the login user.</p>
<tr>
<td align="center">carbon.max.executor.lru.cache.size</td>
<td align="center">NA</td>
-<td align="center">Maximum memory <strong>(in MB)</strong> upto which the executor process can cache the data (DataMaps and reverse dictionary values). Only integer values greater than 0 are accepted. <strong>NOTE:</strong> Mandatory for the user to set.</td>
+<td align="center">Maximum memory <strong>(in MB)</strong> upto which the executor process can cache the data (Indexes and reverse dictionary values). Only integer values greater than 0 are accepted. <strong>NOTE:</strong> Mandatory for the user to set.</td>
</tr>
<tr>
<td align="center">carbon.index.server.max.jobname.length</td>
-<td align="center">NA</td>
+<td align="center">50</td>
<td align="center">The max length of the job to show in the index server application UI. For bigger queries this may impact performance as the whole string would be sent from JDBCServer to IndexServer.</td>
</tr>
<tr>
@@ -425,8 +446,8 @@ executed as a Privileged Action using the login user.</p>
</tr>
<tr>
<td align="center">carbon.indexserver.temp.path</td>
-<td align="center">tablePath</td>
-<td align="center">The folder to write the split files if in memory datamap size for network transfers crossed the 'carbon.index.server.inmemory.serialization.threshold.inKB' limit.</td>
+<td align="center">/tmp/indexservertmp folder</td>
+<td align="center">The folder to write the split files if in memory index cache size for network transfers crossed the 'carbon.index.server.inmemory.serialization.threshold.inKB' limit.</td>
</tr>
</tbody>
</table>
@@ -442,16 +463,6 @@ executed as a Privileged Action using the login user.</p>
</thead>
<tbody>
<tr>
-<td align="center">spark.carbon.indexserver.principal</td>
-<td align="center">NA</td>
-<td align="center">Used for authentication, whether a valid service is trying to connect to the server or not. Set in both IndexServer and JDBCServer.</td>
-</tr>
-<tr>
-<td align="center">spark.carbon.indexserver.keytab</td>
-<td align="center">NA</td>
-<td align="center">Specify the path to the keytab file through which authentication would happen. Set in both IndexServer and JDBCServer.</td>
-</tr>
-<tr>
<td align="center">spark.dynamicAllocation.enabled</td>
<td align="center">true</td>
<td align="center">Set to false, so that spark does not kill the executor, If executors are killed, cache would be lost. Applicable only for Index Server.</td>
@@ -504,6 +515,34 @@ that will authenticate the user to access the index server and no other service.
<td align="center">NA</td>
<td align="center">Set the above property to some appropriate value based on your estimated query time. The best option is to set this to the same value as spark.network.timeout.</td>
</tr>
+<tr>
+<td align="center">hadoop.security.authorization</td>
+<td align="center">false</td>
+<td align="center">Property to enable the hadoop security which is required only on the server side.</td>
+</tr>
+<tr>
+<td align="center">hadoop.proxyuser.<indexserver_user>.users</td>
+<td align="center">NA</td>
+<td align="center">Property to set Proxy User list for which IndexServer permission were to be given ,check <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html" target=_blank rel="nofollow">https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html</a>
+</td>
+</tr>
+<tr>
+<td align="center">hadoop.proxyuser.<indexserver_user>.hosts</td>
+<td align="center">NA</td>
+<td align="center">Property to set hosts list for which IndexServer permission were to be given ,check <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html" target=_blank rel="nofollow">https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html</a>
+</td>
+</tr>
+<tr>
+<td align="center">hadoop.proxyuser.<indexserver_user>.groups</td>
+<td align="center">NA</td>
+<td align="center">Property to set groups list for which IndexServer permission to be given ,check <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html" target=_blank rel="nofollow">https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html</a>
+</td>
+</tr>
+<tr>
+<td align="center">security.indexserver.protocol.acl</td>
+<td align="center">*</td>
+<td align="center">Property to set List of User to be Authorized for Other than proxy Spark Application</td>
+</tr>
</tbody>
</table>
<h5>
@@ -531,10 +570,10 @@ that will authenticate the user to access the index server and no other service.
</table>
<h2>
<a id="starting-the-server" class="anchor" href="#starting-the-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Starting the Server</h2>
-<pre><code>./bin/spark-submit --master [yarn/local] --[optional parameters] --class org.apache.carbondata.indexserver.IndexServer [path to carbondata-spark2-<version>.jar]
+<pre><code>./bin/spark-submit --master [yarn/local] --[optional parameters] --class org.apache.carbondata.indexserver.IndexServer [path to carbondata-spark-<version>.jar]
</code></pre>
<p>Or</p>
-<pre><code>./sbin/start-indexserver.sh --master yarn --num-executors 2 /<absolute path>/carbondata-spark2-1.6.0.0100.jar
+<pre><code>./sbin/start-indexserver.sh --master yarn --num-executors 2 /<absolute path>/carbondata-spark-<version>.jar
</code></pre>
<h2>
<a id="faq" class="anchor" href="#faq" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>FAQ</h2>
@@ -542,9 +581,6 @@ that will authenticate the user to access the index server and no other service.
<p>A. The exception would show the size of response it is trying to send over the
network. Use ipc.maximum.response.length to a value bigger than the
response size.</p>
-<p>Q. <strong>Index server is throwing Kerberos principal not set exception</strong></p>
-<p>A. Set spark.carbon.indexserver.principal to the correct principal in both IndexServer and
-JDBCServer configurations.</p>
<p>Q. <strong>Unable to connect to index server</strong></p>
<p>A. Check whether the carbon.properties configurations are set in JDBCServer as well as the index
server.</p>
diff --git a/content/index.html b/content/index.html
index 5e8bb16..9472247 100644
--- a/content/index.html
+++ b/content/index.html
@@ -54,6 +54,9 @@
class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -75,9 +78,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -322,6 +322,13 @@
</h4>
<div class="linkblock">
<div class="block-row">
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a>
+ <span class="release-date">May 2020</span>
+ <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+2.0.0+Release"
+ class="whatsnew" target="_blank">what's new</a>
+ </div>
+ <div class="block-row">
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a>
<span class="release-date">Oct 2019</span>
@@ -363,13 +370,6 @@
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.5.1+Release"
class="whatsnew" target="_blank">what's new</a>
</div>
- <div class="block-row">
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
- target="_blank">Apache CarbonData 1.5.0</a>
- <span class="release-date">Oct 2018</span>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.5.0+Release"
- class="whatsnew" target="_blank">what's new</a>
- </div>
</div>
<div class="nextR">
<h4 class="title">Release Notes
@@ -501,7 +501,7 @@
to do is:</p>
<ol class="orderlist">
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
target="_blank">Download</a>the latest release.
</li>
diff --git a/content/introduction.html b/content/introduction.html
index 4f27955..8419ed1 100644
--- a/content/introduction.html
+++ b/content/introduction.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -291,32 +300,22 @@
</li>
</ul>
<h3>
-<a id="datamaps" class="anchor" href="#datamaps" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMaps</h3>
+<a id="index" class="anchor" href="#index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index</h3>
<ul>
<li>
<h5>
-<a id="pre-aggregate" class="anchor" href="#pre-aggregate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pre-Aggregate</h5>
-<p>CarbonData has concept of datamaps to assist in pruning of data while querying so that performance is faster.Pre Aggregate tables are kind of datamaps which can improve the query performance by order of magnitude.CarbonData will automatically pre-aggregate the incremental data and re-write the query to automatically fetch from the most appropriate pre-aggregate table to serve the query faster.</p>
-</li>
-<li>
-<h5>
-<a id="time-series" class="anchor" href="#time-series" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Time Series</h5>
-<p>CarbonData has built in understanding of time order(Year, month,day,hour, minute,second). Time series is a pre-aggregate table which can automatically roll-up the data to the desired level during incremental load and serve the query from the most appropriate pre-aggregate table.</p>
-</li>
-<li>
-<h5>
<a id="bloom-filter" class="anchor" href="#bloom-filter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Bloom filter</h5>
-<p>CarbonData supports bloom filter as a datamap in order to quickly and efficiently prune the data for scanning and acheive faster query performance.</p>
+<p>CarbonData supports bloom filter index in order to quickly and efficiently prune the data for scanning and acheive faster query performance.</p>
</li>
<li>
<h5>
<a id="lucene" class="anchor" href="#lucene" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Lucene</h5>
-<p>Lucene is popular for indexing text data which are long.CarbonData provides a lucene datamap so that text columns can be indexed using lucene and use the index result for efficient pruning of data to be retrieved during query.</p>
+<p>Lucene is popular for indexing text data which are long.CarbonData supports lucene index so that text columns can be indexed using lucene and use the index result for efficient pruning of data to be retrieved during query.</p>
</li>
<li>
<h5>
<a id="mv-materialized-views" class="anchor" href="#mv-materialized-views" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MV (Materialized Views)</h5>
-<p>MVs are kind of pre-aggregate tables which can support efficent query re-write and processing.CarbonData provides MV which can rewrite query to fetch from any table(including non-carbondata tables). Typical usecase is to store the aggregated data of a non-carbondata fact table into carbondata and use mv to rewrite the query to fetch from carbondata.</p>
+<p>MVs are kind of pre-aggregate and pre-join tables which can support efficient query re-write and processing.CarbonData provides MV which can rewrite query to fetch from any table(including non-carbondata tables). Typical usecase is to store the aggregated data of a non-carbondata fact table into carbondata and use mv to rewrite the query to fetch from carbondata.</p>
</li>
</ul>
<h3>
diff --git a/content/language-manual.html b/content/language-manual.html
index d874297..331c076 100644
--- a/content/language-manual.html
+++ b/content/language-manual.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -241,15 +250,14 @@
<a href="./ddl-of-carbondata.html">DDL:</a><a href="./ddl-of-carbondata.html#create-table">Create</a>,<a href="./ddl-of-carbondata.html#drop-table">Drop</a>,<a href="./ddl-of-carbondata.html#partition">Partition</a>,<a href="./ddl-of-carbondata.html#bucketing">Bucketing</a>,<a href="./ddl-of-carbondata.html#alter-table">Alter</a>,<a href="./ddl-of-carbondata.html#create-table-as-select">CTAS</a>,<a href="./ddl-of-carbondata.html#create-external-table">External Table</a>
</li>
<li>
-<a href="./datamap-management.html">DataMaps</a>
+<a href="./index/index-management.html">Index</a>
<ul>
-<li><a href="./bloomfilter-datamap-guide.html">Bloom</a></li>
-<li><a href="./lucene-datamap-guide.html">Lucene</a></li>
-<li><a href="./preaggregate-datamap-guide.html">Pre-Aggregate</a></li>
-<li><a href="./timeseries-datamap-guide.html">Time Series</a></li>
+<li><a href="./index/bloomfilter-index-guide.html">Bloom</a></li>
+<li><a href="./index/lucene-index-guide.html">Lucene</a></li>
+<li><a href="./index/secondary-index-guide.html">Secondary-index</a></li>
</ul>
</li>
-<li>Materialized Views (MV)</li>
+<li><a href="./mv-guide.html">Materialized Views</a></li>
<li><a href="./streaming-guide.html">Streaming</a></li>
</ul>
</li>
diff --git a/content/lucene-datamap-guide.html b/content/lucene-index-guide.html
similarity index 77%
rename from content/lucene-datamap-guide.html
rename to content/lucene-index-guide.html
index 534ddf7..6163dc4 100644
--- a/content/lucene-datamap-guide.html
+++ b/content/lucene-index-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,54 +240,56 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-lucene-datamap-alpha-feature" class="anchor" href="#carbondata-lucene-datamap-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Lucene DataMap (Alpha Feature)</h1>
+<a id="carbondata-lucene-index-alpha-feature" class="anchor" href="#carbondata-lucene-index-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Lucene Index (Alpha Feature)</h1>
<ul>
-<li><a href="#datamap-management">DataMap Management</a></li>
-<li><a href="#lucene-datamap-introduction">Lucene Datamap</a></li>
+<li><a href="#index-management">Index Management</a></li>
+<li><a href="#lucene-index-introduction">Lucene Index</a></li>
<li><a href="#loading-data">Loading Data</a></li>
<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#data-management-with-lucene-datamap">Data Management</a></li>
+<li><a href="#data-management-with-lucene-index">Data Management</a></li>
</ul>
<h4>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
-<p>Lucene DataMap can be created using following DDL</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-ON TABLE main_table
-USING 'lucene'
-DMPROPERTIES ('index_columns'='city, name', ...)
+<a id="index-management" class="anchor" href="#index-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Management</h4>
+<p>Lucene Index can be created using following DDL</p>
+<pre><code>CREATE INDEX [IF NOT EXISTS] index_name
+ON TABLE main_table (index_columns)
+AS 'lucene'
+[PROPERTIES ('key'='value')]
</code></pre>
-<p>DataMap can be dropped using following DDL:</p>
-<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
-ON TABLE main_table
+<p>index_columns is the list of string columns on which lucene creates indexes.</p>
+<p>Index can be dropped using following DDL:</p>
+<pre><code>DROP INDEX [IF EXISTS] index_name
+ON [TABLE] main_table
</code></pre>
-<p>To show all DataMaps created, use:</p>
-<pre><code>SHOW DATAMAP
-ON TABLE main_table
+<p>To show all Indexes created, use:</p>
+<pre><code>SHOW INDEXES
+ON [TABLE] main_table
</code></pre>
-<p>It will show all DataMaps created on main table.</p>
+<p>It will show all Indexes created on the main table.</p>
+<blockquote>
+<p>NOTE: Keywords given inside <code>[]</code> is optional.</p>
+</blockquote>
<h2>
-<a id="lucene-datamap-introduction" class="anchor" href="#lucene-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Lucene DataMap Introduction</h2>
+<a id="lucene-index-introduction" class="anchor" href="#lucene-index-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Lucene Index Introduction</h2>
<p>Lucene is a high performance, full featured text search engine. Lucene is integrated to carbon as
-an index datamap and managed along with main tables by CarbonData. User can create lucene datamap
+an index and managed along with main tables by CarbonData. User can create lucene index
to improve query performance on string columns which has content of more length. So, user can
search tokenized word or pattern of it using lucene query on text content.</p>
-<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
-<pre><code>CREATE TABLE datamap_test (
+<p>For instance, main table called <strong>index_test</strong> which is defined as:</p>
+<pre><code>CREATE TABLE index_test (
name string,
age int,
city string,
country string)
STORED AS carbondata
</code></pre>
-<p>User can create Lucene datamap using the Create DataMap DDL:</p>
-<pre><code>CREATE DATAMAP dm
-ON TABLE datamap_test
-USING 'lucene'
-DMPROPERTIES ('INDEX_COLUMNS' = 'name, country',)
+<p>User can create Lucene index using the Create Index DDL:</p>
+<pre><code>CREATE INDEX dm
+ON TABLE index_test (name,country)
+AS 'lucene'
</code></pre>
-<p><strong>DMProperties</strong></p>
+<p><strong>Properties</strong></p>
<ol>
-<li>INDEX_COLUMNS: The list of string columns on which lucene creates indexes.</li>
<li>FLUSH_CACHE: size of the cache to maintain in Lucene writer, if specified then it tries to
aggregate the unique data till the cache limit and flush to Lucene. It is best suitable for low
cardinality dimensions.</li>
@@ -289,78 +300,77 @@ also it makes lucene small chunks of data.</li>
<h2>
<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
<p>When loading data to main table, lucene index files will be generated for all the
-index_columns(String Columns) given in DMProperties which contains information about the data
-location of index_columns. These index files will be written inside a folder named with datamap name
-inside each segment folders.</p>
-<p>A system level configuration carbon.lucene.compression.mode can be added for best compression of
+index_columns(String Columns) given in CREATE statement which contains information about the data
+location of index_columns. These index files will be written inside a folder named with index name
+inside each segment folder.</p>
+<p>A system level configuration <code>carbon.lucene.compression.mode</code> can be added for best compression of
lucene index files. The default value is speed, where the index writing speed will be more. If the
value is compression, the index file size will be compressed.</p>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
<p>As a technique for query acceleration, Lucene indexes cannot be queried directly.
-Queries are to be made on main table. when a query with TEXT_MATCH('name:c10') or
+Queries are to be made on the main table. When a query with TEXT_MATCH('name:c10') or
TEXT_MATCH_WITH_LIMIT('name:n10',10)[the second parameter represents the number of result to be
returned, if user does not specify this value, all results will be returned without any limit] is
-fired, two jobs are fired. The first job writes the temporary files in folder created at table level
-which contains lucene's seach results and these files will be read in second job to give faster
+fired, two jobs will be launched. The first job writes the temporary files in folder created at table level
+which contains lucene's search results and these files will be read in second job to give faster
results. These temporary files will be cleared once the query finishes.</p>
-<p>User can verify whether a query can leverage Lucene datamap or not by executing <code>EXPLAIN</code>
+<p>User can verify whether a query can leverage Lucene index or not by executing the <code>EXPLAIN</code>
command, which will show the transformed logical plan, and thus user can check whether TEXT_MATCH()
filter is applied on query or not.</p>
<p><strong>Note:</strong></p>
<ol>
<li>
-<p>The filter columns in TEXT_MATCH or TEXT_MATCH_WITH_LIMIT must be always in lower case and
-filter condition like 'AND','OR' must be in upper case.</p>
+<p>The filter columns in TEXT_MATCH or TEXT_MATCH_WITH_LIMIT must be always in lowercase and
+filter conditions like 'AND','OR' must be in upper case.</p>
<p>Ex:</p>
-<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+<pre><code>select * from index_test where TEXT_MATCH('name:*10 AND name:*n*')
</code></pre>
</li>
<li>
<p>Query supports only one TEXT_MATCH udf for filter condition and not multiple udfs.</p>
<p>The following query is supported:</p>
-<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+<pre><code>select * from index_test where TEXT_MATCH('name:*10 AND name:*n*')
</code></pre>
<p>The following query is not supported:</p>
-<pre><code>select * from datamap_test where TEXT_MATCH('name:*10) AND TEXT_MATCH(name:*n*')
+<pre><code>select * from index_test where TEXT_MATCH('name:*10) AND TEXT_MATCH(name:*n*')
</code></pre>
</li>
</ol>
-<p>Below like queries can be converted to text_match queries as following:</p>
-<pre><code>select * from datamap_test where name='n10'
+<p>Below <code>like</code> queries can be converted to text_match queries as following:</p>
+<pre><code>select * from index_test where name='n10'
-select * from datamap_test where name like 'n1%'
+select * from index_test where name like 'n1%'
-select * from datamap_test where name like '%10'
+select * from index_test where name like '%10'
-select * from datamap_test where name like '%n%'
+select * from index_test where name like '%n%'
-select * from datamap_test where name like '%10' and name not like '%n%'
+select * from index_test where name like '%10' and name not like '%n%'
</code></pre>
<p>Lucene TEXT_MATCH Queries:</p>
-<pre><code>select * from datamap_test where TEXT_MATCH('name:n10')
+<pre><code>select * from index_test where TEXT_MATCH('name:n10')
-select * from datamap_test where TEXT_MATCH('name:n1*')
+select * from index_test where TEXT_MATCH('name:n1*')
-select * from datamap_test where TEXT_MATCH('name:*10')
+select * from index_test where TEXT_MATCH('name:*10')
-select * from datamap_test where TEXT_MATCH('name:*n*')
+select * from index_test where TEXT_MATCH('name:*n*')
-select * from datamap_test where TEXT_MATCH('name:*10 -name:*n*')
+select * from index_test where TEXT_MATCH('name:*10 -name:*n*')
</code></pre>
<p><strong>Note:</strong> For lucene queries and syntax, refer to <a href="http://www.lucenetutorial.com/lucene-query-syntax.html" target=_blank rel="nofollow">lucene-syntax</a></p>
<h2>
-<a id="data-management-with-lucene-datamap" class="anchor" href="#data-management-with-lucene-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with lucene datamap</h2>
-<p>Once there is lucene datamap is created on the main table, following command on the main
-table
-is not supported:</p>
+<a id="data-management-with-lucene-index" class="anchor" href="#data-management-with-lucene-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with lucene index</h2>
+<p>Once there is a lucene index created on the main table, following command on the main
+table is not supported:</p>
<ol>
<li>Data management command: <code>UPDATE/DELETE</code>.</li>
<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
<code>ALTER TABLE RENAME</code>.</li>
</ol>
<p><strong>Note</strong>: Adding a new column is supported, and for dropping columns and change datatype
-command, CarbonData will check whether it will impact the lucene datamap, if not, the operation
+command, CarbonData will check whether it will impact the lucene index, if not, the operation
is allowed, otherwise operation will be rejected by throwing exception.</p>
<ol start="3">
<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>.</li>
@@ -368,10 +378,10 @@ is allowed, otherwise operation will be rejected by throwing exception.</p>
<p>However, there is still way to support these operations on main table, in current CarbonData
release, user can do as following:</p>
<ol>
-<li>Remove the lucene datamap by <code>DROP DATAMAP</code> command.</li>
+<li>Remove the lucene index by <code>DROP INDEX</code> command.</li>
<li>Carry out the data management operation on main table.</li>
-<li>Create the lucene datamap again by <code>CREATE DATAMAP</code> command.
-Basically, user can manually trigger the operation by re-building the datamap.</li>
+<li>Create the lucene index again by <code>CREATE INDEX</code> command.
+Basically, user can manually trigger the operation by refreshing the index.</li>
</ol>
<script>
$(function() {
diff --git a/content/mv-datamap-guide.html b/content/mv-guide.html
similarity index 50%
copy from content/mv-datamap-guide.html
copy to content/mv-guide.html
index 22cc201..bddeb5e 100644
--- a/content/mv-datamap-guide.html
+++ b/content/mv-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,63 +240,60 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-mv-datamap" class="anchor" href="#carbondata-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData MV DataMap</h1>
+<a id="carbondata-materialized-view" class="anchor" href="#carbondata-materialized-view" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Materialized View</h1>
<ul>
<li><a href="#quick-example">Quick Example</a></li>
-<li><a href="#mv-datamap-introduction">MV DataMap</a></li>
+<li><a href="#introduction">Introduction</a></li>
<li><a href="#loading-data">Loading Data</a></li>
<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#compacting-mv-datamap">Compaction</a></li>
-<li><a href="#data-management-with-mv-tables">Data Management</a></li>
+<li><a href="#compacting">Compaction</a></li>
+<li><a href="#data-management">Data Management</a></li>
+<li><a href="#time-series-support">Time Series Support</a></li>
+<li><a href="#time-series-rollup-support">Time Series RollUp Support</a></li>
</ul>
<h2>
<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
<p>Start spark-sql in terminal and run the following queries,</p>
-<pre><code>CREATE TABLE maintable(a int, b string, c int) stored by 'carbondata';
-insert into maintable select 1, 'ab', 2;
-CREATE DATAMAP datamap_1 on table maintable as SELECT a, sum(b) from maintable group by a;
-SELECT a, sum(b) from maintable group by a;
-// NOTE: run explain query and check if query hits the datamap table from the plan
-EXPLAIN SELECT a, sum(b) from maintable group by a;
+<pre><code> CREATE TABLE maintable(a int, b string, c int) stored as carbondata;
+ INSERT INTO maintable SELECT 1, 'ab', 2;
+ CREATE MATERIALIZED VIEW view1 AS SELECT a, sum(b) FROM maintable GROUP BY a;
+ SELECT a, sum(b) FROM maintable GROUP BY a;
+ // NOTE: run explain query and check if query hits the mv table from the plan
+ EXPLAIN SELECT a, sum(b) FROM maintable GROUP BY a;
</code></pre>
<h2>
-<a id="mv-datamap-introduction" class="anchor" href="#mv-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MV DataMap Introduction</h2>
-<p>MV tables are created as DataMaps and managed as tables internally by CarbonData. User can create
-limitless MV datamaps on a table to improve query performance provided the storage requirements
-and loading time is acceptable.</p>
-<p>MV datamap can be a lazy or a non-lazy datamap. Once MV datamaps are created, CarbonData's
-CarbonAnalyzer helps to select the most efficient MV datamap based on the user query and rewrite
-the SQL to select the data from MV datamap instead of main table. Since the data size of MV
-datamap is smaller and data is pre-processed, user queries are much faster.</p>
-<p>For instance, main table called <strong>sales</strong> which is defined as</p>
-<pre><code>CREATE TABLE sales (
- order_time timestamp,
- user_id string,
- sex string,
- country string,
- quantity int,
- price bigint)
-STORED AS carbondata
+<a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Introduction</h2>
+<p>Materialized views are created as tables from queries. Users can create limitless materialized views
+to improve query performance provided the storage requirements and loading time is acceptable.</p>
+<p>Materialized view can be refreshed on commit or on manual. Once materialized views are created,
+CarbonData's <code>MVRewriteRule</code> helps to select the most efficient materialized view based on
+the user query and rewrite the SQL to select the data from materialized view instead of
+fact tables. Since the data size of materialized view is smaller and data is pre-processed,
+user queries are much faster.</p>
+<p>For instance, fact table called <strong>sales</strong> which is defined as.</p>
+<pre><code> CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+ STORED AS carbondata
</code></pre>
-<p>User can create MV tables using the Create DataMap DDL</p>
-<pre><code>CREATE DATAMAP agg_sales
-ON TABLE sales
-USING "MV"
-DMPROPERTIES('TABLE_BLOCKSIZE'='256 MB','LOCAL_DICTIONARY_ENABLE'='false')
-AS
- SELECT country, sex, sum(quantity), avg(price)
- FROM sales
- GROUP BY country, sex
+<p>Users can create a materialized view using the CREATE MATERIALIZED VIEW statement.</p>
+<pre><code> CREATE MATERIALIZED VIEW agg_sales
+ PROPERTIES('TABLE_BLOCKSIZE'='256 MB','LOCAL_DICTIONARY_ENABLE'='false')
+ AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
</code></pre>
<p><strong>NOTE</strong>:</p>
<ul>
-<li>
-<p>Group by/Filter columns has to be provided in projection list while creating mv datamap</p>
-</li>
-<li>
-<p>If only single parent table is involved in mv datamap creation, then TableProperties of Parent table
-(if not present in a aggregate function like sum(col)) listed below will be
-inherited to datamap table</p>
+<li>Group by and Order by columns has to be provided in the projection list while creating a materialized view.</li>
+<li>If only single fact table is involved in materialized view creation, then TableProperties of
+fact table (if not present in a aggregate function like sum(col)) listed below will be
+inherited to materialized view.
<ol>
<li>SORT_COLUMNS</li>
<li>SORT_SCOPE</li>
@@ -297,143 +303,262 @@ inherited to datamap table</p>
<li>LOCAL_DICTIONARY_ENABLE</li>
<li>LOCAL_DICTIONARY_THRESHOLD</li>
<li>LOCAL_DICTIONARY_EXCLUDE</li>
-<li>DICTIONARY_INCLUDE</li>
-<li>DICTIONARY_EXCLUDE</li>
<li>INVERTED_INDEX</li>
<li>NO_INVERTED_INDEX</li>
<li>COLUMN_COMPRESSOR</li>
</ol>
</li>
-<li>
-<p>All columns of main table at once cannot participate in mv datamap table creation</p>
-</li>
-<li>
-<p>TableProperties can be provided in DMProperties excluding LOCAL_DICTIONARY_INCLUDE,
-LOCAL_DICTIONARY_EXCLUDE, DICTIONARY_INCLUDE, DICTIONARY_EXCLUDE, INVERTED_INDEX,
-NO_INVERTED_INDEX, SORT_COLUMNS, LONG_STRING_COLUMNS, RANGE_COLUMN & COLUMN_META_CACHE</p>
-</li>
-<li>
-<p>TableProperty given in DMProperties will be considered for mv creation, eventhough if same
-property is inherited from parent table, which allows user to provide different tableproperties
-for child table</p>
-</li>
-<li>
-<p>MV creation with limit or union all ctas queries is unsupported</p>
-</li>
-<li>
-<p>MV datamap does not support Streaming</p>
+<li>Creating materialized view with select query containing only project of all columns of fact
+table is unsupported.
+<strong>Example:</strong>
+If table 'x' contains columns 'a,b,c', then creating MV with below queries is not supported.
+1. <code>SELECT a,b,c FROM x</code>
+2. <code>SELECT * FROM x</code>
</li>
+<li>TableProperties can be provided in Properties excluding LOCAL_DICTIONARY_INCLUDE,
+LOCAL_DICTIONARY_EXCLUDE, INVERTED_INDEX, NO_INVERTED_INDEX, SORT_COLUMNS, LONG_STRING_COLUMNS,
+RANGE_COLUMN & COLUMN_META_CACHE.</li>
+<li>TableProperty given in Properties will be considered for materialized view creation, even though
+if same property is inherited from fact table, which allows user to provide different table
+properties for materialized view.</li>
+<li>Materialized view creation with limit or union all CTAS queries is unsupported.</li>
+<li>Materialized view does not support streaming.</li>
</ul>
<h4>
-<a id="how-mv-tables-are-selected" class="anchor" href="#how-mv-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How MV tables are selected</h4>
-<p>When a user query is submitted, during query planning phase, CarbonData will collect modular plan
-candidates and process the the ModularPlan based on registered summary data sets. Then,
-mv datamap table for this query will be selected among the candidates.</p>
-<p>For the main table <strong>sales</strong> and mv table <strong>agg_sales</strong> created above, following queries</p>
-<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
-
-SELECT sex, sum(quantity) from sales GROUP BY sex
-
-SELECT avg(price), country from sales GROUP BY country
+<a id="how-materialized-views-are-selected" class="anchor" href="#how-materialized-views-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How materialized views are selected</h4>
+<p>When a user query is submitted, during the query planning phase, CarbonData will collect modular plan
+candidates and process the ModularPlan based on registered summary data sets. Then,
+a materialized view for this query will be selected among the candidates.</p>
+<p>For the fact table <strong>sales</strong> and materialized view <strong>agg_sales</strong> created above, following queries</p>
+<pre><code> SELECT country, sex, sum(quantity), avg(price) FROM sales GROUP BY country, sex
+ SELECT sex, sum(quantity) FROM sales GROUP BY sex
+ SELECT avg(price), country FROM sales GROUP BY country
</code></pre>
-<p>will be transformed by CarbonData's query planner to query against mv table
-<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
+<p>will be transformed by CarbonData's query planner to query against materialized view <strong>agg_sales</strong>
+instead of the fact table <strong>sales</strong>.</p>
<p>However, for following queries</p>
-<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
-
-SELECT sex, avg(quantity) from sales GROUP BY sex
-
-SELECT country, max(price) from sales GROUP BY country
+<pre><code> SELECT user_id, country, sex, sum(quantity), avg(price) FROM sales GROUP BY user_id, country, sex
+ SELECT sex, avg(quantity) FROM sales GROUP BY sex
+ SELECT country, max(price) FROM sales GROUP BY country
</code></pre>
-<p>will query against main table <strong>sales</strong> only, because it does not satisfy mv table
+<p>will query against fact table <strong>sales</strong> only, because it does not satisfy materialized view
selection logic.</p>
<h2>
<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
<h3>
-<a id="loading-data-to-non-lazy-mv-datamap" class="anchor" href="#loading-data-to-non-lazy-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Non-Lazy MV Datamap</h3>
-<p>In case of WITHOUT DEFERRED REBUILD, for existing table with loaded data, data load to MV table will
-be triggered by the CREATE DATAMAP statement when user creates the MV table.
-For incremental loads to main table, data to datamap will be loaded once the corresponding main
-table load is completed.</p>
+<a id="loading-data-on-commit" class="anchor" href="#loading-data-on-commit" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data on commit</h3>
+<p>In case of WITHOUT DEFERRED REFRESH, for existing table with loaded data, data load to materialized
+view will be triggered by the CREATE MATERIALIZED VIEW statement when user creates the materialized
+view.</p>
+<p>For incremental loads to the fact table, data to materialized view will be loaded once the
+corresponding fact table load is completed.</p>
<h3>
-<a id="loading-data-to-lazy-mv-datamap" class="anchor" href="#loading-data-to-lazy-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Lazy MV Datamap</h3>
-<p>In case of WITH DEFERRED REBUILD, data load to MV table will be triggered by the <a href="./datamap-management.html#manual-refresh">Manual Refresh</a>
-command. MV datamap will be in DISABLED state in below scenarios,</p>
+<a id="loading-data-on-manual" class="anchor" href="#loading-data-on-manual" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data on manual</h3>
+<p>In case of WITH DEFERRED REFRESH, data load to materialized view will be triggered by the refresh
+command. Materialized view will be in DISABLED state in below scenarios.</p>
<ul>
-<li>when mv datamap is created</li>
-<li>when data of main table and datamap are not in sync</li>
+<li>when a materialized view is created.</li>
+<li>when data of fact table and materialized view are not in sync.</li>
</ul>
-<p>User should fire REBUILD DATAMAP command to sync all segments of main table with datamap table and
-which ENABLES the datamap for query</p>
+<p>User should fire REFRESH MATERIALIZED VIEW command to sync all segments of fact table with
+materialized view, which ENABLES the materialized view for query.</p>
+<p>Command example:</p>
+<pre><code> REFRESH MATERIALIZED VIEW agg_sales
+</code></pre>
<h3>
-<a id="loading-data-to-multiple-mvs" class="anchor" href="#loading-data-to-multiple-mvs" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Multiple MV's</h3>
-<p>During load to main table, if anyone of the load to datamap table fails, then that corresponding
-datamap will be DISABLED and load to other datamaps mapped to main table will continue. User can
-fire REBUILD DATAMAP command to sync or else the subsequent table load will load the old failed
-loads along with current load and enable the disabled datamap.</p>
+<a id="loading-data-to-multiple-materialized-views" class="anchor" href="#loading-data-to-multiple-materialized-views" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to multiple materialized views</h3>
+<p>During load to fact table, if anyone of the load to materialized view fails, then that
+corresponding materialized view will be DISABLED and load to other materialized views mapped
+to the fact table will continue.</p>
+<p>User can fire REFRESH MATERIALIZED VIEW command to sync or else the subsequent table load
+will load the old failed loads along with current load and enable the disabled materialized view.</p>
<p><strong>NOTE</strong>:</p>
<ul>
-<li>In case of InsertOverwrite/Update operation on parent table, all segments of datamap table will
-be MARKED_FOR_DELETE and reload to datamap table will happen by REBUILD DATAMAP, in case of Lazy
-mv datamap/ once InsertOverwrite/Update operation on parent table is finished, in case of
-Non-Lazy mv.</li>
-<li>In case of full scan query, Data Size and Index Size of main table and child table will not the
-same, as main table and child table has different column names.</li>
+<li>In case of InsertOverwrite/Update operation on fact table, all segments of materialized view
+will be MARKED_FOR_DELETE and reload to mv table will happen by REFRESH MATERIALIZED VIEW,
+in case of materialized view which refresh on manual and once the InsertOverwrite/Update
+operation on fact table is finished, in case of materialized view which refresh on commit.</li>
+<li>In case of full scan query, Data Size and Index Size of fact table and materialized view
+will not be the same, as fact table and materialized view have different column names.</li>
</ul>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
-<p>As a technique for query acceleration, MV tables cannot be queried directly.
-Queries are to be made on main table. While doing query planning, internally CarbonData will check
-associated mv datamap tables with the main table, and do query plan transformation accordingly.</p>
-<p>User can verify whether a query can leverage mv datamap table or not by executing <code>EXPLAIN</code>
-command, which will show the transformed logical plan, and thus user can check whether mv datamap
-table is selected.</p>
+<p>Queries are to be made on the fact table. While doing query planning, internally CarbonData will check
+for the materialized views which are associated with the fact table, and do query plan
+transformation accordingly.</p>
+<p>Users can verify whether a query can leverage materialized view or not by executing the <code>EXPLAIN</code> command,
+which will show the transformed logical plan, and thus the user can check whether a materialized view
+is selected.</p>
<h2>
-<a id="compacting-mv-datamap" class="anchor" href="#compacting-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap</h2>
-<h3>
-<a id="compacting-mv-datamap-table-through-main-table-compaction" class="anchor" href="#compacting-mv-datamap-table-through-main-table-compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap table through Main Table compaction</h3>
-<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>)[COMPACTION TYPE-> MINOR/MAJOR] on main table will
-automatically compact the mv datamap tables created on the main table, once compaction on main table
-is done.</p>
-<h3>
-<a id="compacting-mv-datamap-table-through-ddl-command" class="anchor" href="#compacting-mv-datamap-table-through-ddl-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap table through DDL command</h3>
-<p>Compaction on mv datamap can be triggered by running the following DDL command(supported only for mv).</p>
-<pre><code>ALTER DATAMAP datamap_name COMPACT 'COMPACTION_TYPE'
-</code></pre>
+<a id="compacting" class="anchor" href="#compacting" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting</h2>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>)[COMPACTION TYPE-> MINOR/MAJOR] on fact table
+will automatically compact the materialized view created on the fact table, once compaction
+on fact table is done.</p>
<h2>
-<a id="data-management-with-mv-tables" class="anchor" href="#data-management-with-mv-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with mv tables</h2>
-<p>In current implementation, data consistency needs to be maintained for both main table and mv datamap
-tables. Once there is mv datamap table created on the main table, following command on the main
+<a id="data-management" class="anchor" href="#data-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management</h2>
+<p>In current implementation, data consistency needs to be maintained for both fact table and
+materialized views.</p>
+<p>Once there is materialized view created on the fact table, following command on the fact
table is not supported:</p>
<ol>
<li>Data management command: <code>DELETE SEGMENT</code>.</li>
<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
<code>ALTER TABLE RENAME</code>, <code>ALTER COLUMN RENAME</code>. Note that adding a new column is supported, and for
dropping columns and change datatype command, CarbonData will check whether it will impact the
-mv datamap table, if not, the operation is allowed, otherwise operation will be rejected by
+materialized view, if not, the operation is allowed, otherwise operation will be rejected by
throwing exception.</li>
<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>. Note that dropping a partition
-will be allowed only if partition is participating in all datamaps associated with main table.
-Drop Partition is not allowed, if any mv datamap is associated with more than one parent table.
-Drop Partition directly on datamap table is not allowed.</li>
-<li>Complex Datatype's for mv datamap is not supported.</li>
+will be allowed only if the partition column of fact table is participating in all of the table's materialized views.
+Drop Partition is not allowed, if any materialized view is associated with more than one
+fact table. Drop Partition directly on materialized view is not allowed.</li>
+<li>Complex Datatype's for materialized view is not supported.</li>
</ol>
-<p>However, there is still way to support these operations on main table, in current CarbonData
+<p>However, there is still way to support these operations on fact table, in current CarbonData
release, user can do as following:</p>
<ol>
-<li>Remove the mv datamap table by <code>DROP DATAMAP</code> command</li>
-<li>Carry out the data management operation on main table</li>
-<li>Create the mv datamap table again by <code>CREATE DATAMAP</code> command
-Basically, user can manually trigger the operation by re-building the datamap.</li>
+<li>Remove the materialized view by <code>DROP MATERIALIZED VIEW</code> command.</li>
+<li>Carry out the data management operation on fact table.</li>
+<li>Create the materialized view again by <code>CREATE MATERIALIZED VIEW</code> command.</li>
+</ol>
+<p>Basically, user can manually trigger the operation by re-building the materialized view.</p>
+<h2>
+<a id="time-series-support" class="anchor" href="#time-series-support" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Time Series Support</h2>
+<p>Time series data are simply measurements or events that are tracked, monitored, down sampled, and
+aggregated over time. Materialized views with automatic refresh mode supports TimeSeries queries.</p>
+<p>CarbonData provides built-in time-series udf with the below definition.</p>
+<pre><code> timeseries(event_time_column, 'granularity')
+</code></pre>
+<p>Event time columns provided in time series udf should be of TimeStamp/Date type.</p>
+<p>Below table describes the time hierarchy and levels that can be provided in a time-series udf,
+so that it supports automatic roll-up in time dimension for query.</p>
+<table>
+<thead>
+<tr>
+<th>Granularity</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>year</td>
+<td>Data will be aggregated over year</td>
+</tr>
+<tr>
+<td>month</td>
+<td>Data will be aggregated over month</td>
+</tr>
+<tr>
+<td>week</td>
+<td>Data will be aggregated over week</td>
+</tr>
+<tr>
+<td>day</td>
+<td>Data will be aggregated over day</td>
+</tr>
+<tr>
+<td>hour</td>
+<td>Data will be aggregated over hour</td>
+</tr>
+<tr>
+<td>thirty_minute</td>
+<td>Data will be aggregated over every thirty minutes</td>
+</tr>
+<tr>
+<td>fifteen_minute</td>
+<td>Data will be aggregated over every fifteen minutes</td>
+</tr>
+<tr>
+<td>ten_minute</td>
+<td>Data will be aggregated over every ten minutes</td>
+</tr>
+<tr>
+<td>five_minute</td>
+<td>Data will be aggregated over every five minutes</td>
+</tr>
+<tr>
+<td>minute</td>
+<td>Data will be aggregated over every one minute</td>
+</tr>
+<tr>
+<td>second</td>
+<td>Data will be aggregated over every second</td>
+</tr>
+</tbody>
+</table>
+<p>Time series udf having column as Date type support's only year, month, day and week granularities.</p>
+<p>Below is the sample data loaded to the fact table <strong>sales</strong>.</p>
+<pre><code> order_time, user_id, sex, country, quantity, price
+ 2016-02-23 09:01:30, c001, male, xxx, 100, 2
+ 2016-02-23 09:01:50, c002, male, yyy, 200, 5
+ 2016-02-23 09:03:30, c003, female, xxx, 400, 1
+ 2016-02-23 09:03:50, c004, male, yyy, 300, 5
+ 2016-02-23 09:07:50, c005, female, xxx, 500, 5
+</code></pre>
+<p>Users can create materialized views with time series queries like the below example:</p>
+<pre><code> CREATE MATERIALIZED VIEW agg_sales AS
+ SELECT timeseries(order_time, 'minute'),avg(price)
+ FROM sales
+ GROUP BY timeseries(order_time, 'minute')
+</code></pre>
+<p>And execute the below query to check time series data. In this example, a materialized view of
+the aggregated table on the price column will be created, which will be aggregated every one minute.</p>
+<pre><code> SELECT timeseries(order_time,'minute'), avg(price)
+ FROM sales
+ GROUP BY timeseries(order_time,'minute')
+</code></pre>
+<p>Find below the result of the above query aggregated over a minute.</p>
+<pre><code> +---------------------------------------+----------------+
+ |UDF:timeseries(order_time, minute) |avg(price) |
+ +---------------------------------------+----------------+
+ |2016-02-23 09:01:00 |3.5 |
+ |2016-02-23 09:07:00 |5.0 |
+ |2016-02-23 09:03:00 |3.0 |
+ +---------------------------------------+----------------+
+</code></pre>
+<p>The data loading, querying, compaction command and its behavior is the same as materialized views.</p>
+<h4>
+<a id="how-data-is-aggregated-over-time" class="anchor" href="#how-data-is-aggregated-over-time" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How data is aggregated over time?</h4>
+<p>On each load to materialized view, data will be aggregated based on the specified time interval of
+granularity provided during creation and stored on each segment.</p>
+<p><strong>NOTE</strong>:</p>
+<ol>
+<li>Retention policies for time series is not supported yet.</li>
</ol>
+<h2>
+<a id="time-series-rollup-support" class="anchor" href="#time-series-rollup-support" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Time Series RollUp Support</h2>
+<p>Time series queries can be rolled up from an existing materialized view.</p>
+<h3>
+<a id="query-rollup" class="anchor" href="#query-rollup" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query RollUp</h3>
+<p>Consider an example where the query is on hour level granularity, but the materialized view
+with hour level granularity is not present but materialized view with minute level granularity is
+present, then we can get the data from minute level and aggregate the hour level data and
+give output. This is called query rollup.</p>
+<p>Consider if user create's below time series materialized view,</p>
+<pre><code> CREATE MATERIALIZED VIEW agg_sales
+ AS
+ SELECT timeseries(order_time,'minute'),avg(price)
+ FROM sales
+ GROUP BY timeseries(order_time,'minute')
+</code></pre>
+<p>and fires the below query with hour level granularity.</p>
+<pre><code> SELECT timeseries(order_time,'hour'),avg(price)
+ FROM sales
+ GROUP BY timeseries(order_time,'hour')
+</code></pre>
+<p>Then, the above query can be rolled up from materialized view 'agg_sales', by adding hour
+level time series aggregation on minute level aggregation. Users can fire the <code>EXPLAIN</code> command
+to check if a query is rolled up from an existing materialized view.</p>
+<p><strong>NOTE</strong>:
+1. Queries cannot be rolled up, if the filter contains a time series function.
+2. Roll up is not yet supported for queries having join clause or order by functions.</p>
<script>
$(function() {
// Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
+ $('.b-nav__docs').addClass('selected');
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ // Display docs subnav items
+ if (!$('.b-nav__docs').parent().hasClass('nav__item__with__subs--expanded')) {
+ $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
}
});
</script></div>
diff --git a/content/performance-tuning.html b/content/performance-tuning.html
index 2ec174c..8822b4e 100644
--- a/content/performance-tuning.html
+++ b/content/performance-tuning.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -317,7 +326,7 @@ The create table command can be modified as suggested below :</p>
BEGIN_TIME bigint,
HOST String,
Dime_1 String,
- counter_1, Decimal
+ counter_1 Decimal,
...
)STORED AS carbondata
@@ -338,7 +347,7 @@ The create table command can be modified as suggested below :</p>
BEGIN_TIME bigint,
HOST String,
Dime_1 String,
- counter_1, Decimal
+ counter_1 Decimal,
...
)STORED AS carbondata
@@ -382,7 +391,10 @@ Incremental values are efficient in using min/max index. The create table comman
</code></pre>
<p><strong>NOTE:</strong></p>
<ul>
-<li>BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter datamap <a href="./bloomfilter-datamap-guide.html">document</a>.</li>
+<li>BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter index <a href="./index/bloomfilter-index-guide.html">document</a>.</li>
+<li>Lucene index can be created on string columns which has content of more length to enhance the query performance. You can find more information about it in Lucene index <a href="./index/lucene-index-guide.html">document</a>.</li>
+<li>Secondary index can be created based on the column position in main table(Recommended for right columns) and the queries should have filter on that column to improve the filter query performance. You can find more information about it in secondary index <a href="./index/secondary-index-guide.html">document</a>.</li>
+<li>Materialized view can be created to improve query performance provided the storage requirements and loading time is acceptable. You can find more information about it in materialized view <a href="./mv-guide.html">document</a>.</li>
</ul>
<h2>
<a id="configuration-for-optimizing-data-loading-performance-for-massive-data" class="anchor" href="#configuration-for-optimizing-data-loading-performance-for-massive-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuration for Optimizing Data Loading performance for Massive Data</h2>
@@ -406,12 +418,12 @@ You can configure CarbonData by tuning following properties in carbon.properties
</tr>
<tr>
<td>carbon.sort.size</td>
-<td>Default: 100000. The value should be >= 100.</td>
+<td>Default: 100000. The value should be >= 1000.</td>
<td>Threshold to write local file in sort step when loading data</td>
</tr>
<tr>
<td>carbon.sort.file.write.buffer.size</td>
-<td>Default: 16384.</td>
+<td>Default: 16384. The value should be >= 10240 and <= 10485760.</td>
<td>CarbonData sorts and writes data to intermediate files to limit the memory usage. This configuration determines the buffer size to be used for reading and writing such files.</td>
</tr>
<tr>
@@ -426,7 +438,7 @@ You can configure CarbonData by tuning following properties in carbon.properties
</tr>
</tbody>
</table>
-<p>For example, if there are 10 million records, and i have only 16 cores, 64GB memory, will be loaded to CarbonData table.
+<p>For example, if there are 10 million records, and I have only 16 cores, 64 GB memory will be loaded to CarbonData table.
Using the default configuration always fail in sort step. Modify carbon.properties as suggested below:</p>
<pre><code>carbon.merge.sort.reader.thread=1
carbon.sort.size=5000
@@ -511,13 +523,6 @@ scenarios. After the completion of POC, some of the configurations impacting the
<td>Whether to enable size based block allocation strategy for data loading.</td>
<td>When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB to 1GB.</td>
</tr>
-<tr>
-<td>carbon.load.min.size.enabled</td>
-<td>spark/carbonlib/carbon.properties</td>
-<td>Data loading</td>
-<td>Whether to enable node minumun input data size allocation strategy for data loading.</td>
-<td>When loading, carbondata will use node minumun input data size allocation strategy for task distribution. It will make sure the nodes load the minimum amount of data -- It's useful if the size of your input data files very small, say 1MB to 256MB,Avoid generating a large number of small files.</td>
-</tr>
</tbody>
</table>
<p>Note: If your CarbonData instance is provided only for query, you may specify the property 'spark.speculation=true' which is in conf directory of spark.</p>
diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html
deleted file mode 100644
index 91d561e..0000000
--- a/content/preaggregate-datamap-guide.html
+++ /dev/null
@@ -1,519 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
- <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
- <title>CarbonData</title>
- <style>
-
- </style>
- <!-- Bootstrap -->
-
- <link rel="stylesheet" href="css/bootstrap.min.css">
- <link href="css/style.css" rel="stylesheet">
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- <script src="js/jquery.min.js"></script>
- <script src="js/bootstrap.min.js"></script>
- <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
-
-
-</head>
-<body>
-<header>
- <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
- <div class="container">
- <div class="navbar-header">
- <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
- class="navbar-toggle collapsed" type="button">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a href="index.html" class="logo">
- <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
- </a>
- </div>
- <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
- <ul class="nav navbar-nav navbar-right navlist-custom">
- <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
- </li>
- <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false"> Download <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
- target="_blank">Apache CarbonData 1.6.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.0/"
- target="_blank">Apache CarbonData 1.6.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.4/"
- target="_blank">Apache CarbonData 1.5.4</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.3/"
- target="_blank">Apache CarbonData 1.5.3</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.2/"
- target="_blank">Apache CarbonData 1.5.2</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"
- target="_blank">Apache CarbonData 1.5.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
- target="_blank">Apache CarbonData 1.5.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
- target="_blank">Release Archive</a></li>
- </ul>
- </li>
- <li><a href="documentation.html" class="active">Documentation</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false">Community <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
- target="_blank">Contributing to CarbonData</a></li>
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
- target="_blank">Release Guide</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
- target="_blank">Project PMC and Committers</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
- target="_blank">CarbonData Meetups</a></li>
- <li><a href="security.html">Apache CarbonData Security</a></li>
- <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
- Jira</a></li>
- <li><a href="videogallery.html">CarbonData Videos </a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li class="dropdown">
- <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li>
- <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
-
- </li>
-
- </ul>
- </div><!--/.nav-collapse -->
- <div id="search-box">
- <form method="get" action="http://www.google.com/search" target="_blank">
- <div class="search-block">
- <table border="0" cellpadding="0" width="100%">
- <tr>
- <td style="width:80%">
- <input type="text" name="q" size=" 5" maxlength="255" value=""
- class="search-input" placeholder="Search...." required/>
- </td>
- <td style="width:20%">
- <input type="submit" value="Search"/></td>
- </tr>
- <tr>
- <td align="left" style="font-size:75%" colspan="2">
- <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
- <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
- </td>
- </tr>
- </table>
- </div>
- </form>
- </div>
- </div>
- </nav>
-</header> <!-- end Header part -->
-
-<div class="fixed-padding"></div> <!-- top padding with fixde header -->
-
-<section><!-- Dashboard nav -->
- <div class="container-fluid q">
- <div class="col-sm-12 col-md-12 maindashboard">
- <div class="verticalnavbar">
- <nav class="b-sticky-nav">
- <div class="nav-scroller">
- <div class="nav__inner">
- <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
- <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
- <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
- <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
- <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
- <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
- <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
- <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
- <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
- <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
- <a class="nav__item nav__sub__item" href="./csdk-guide.html">C++ SDK</a>
- </div>
-
- <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
- <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
- <a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
- <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
- <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
- <a class="b-nav__security nav__item" href="./security.html">Security</a>
- <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
- </div>
- </div>
- <div class="navindicator">
- <div class="b-nav__intro navindicator__item"></div>
- <div class="b-nav__quickstart navindicator__item"></div>
- <div class="b-nav__uses navindicator__item"></div>
- <div class="b-nav__docs navindicator__item"></div>
- <div class="b-nav__datamap navindicator__item"></div>
- <div class="b-nav__api navindicator__item"></div>
- <div class="b-nav__perf navindicator__item"></div>
- <div class="b-nav__s3 navindicator__item"></div>
- <div class="b-nav__indexserver navindicator__item"></div>
- <div class="b-nav__faq navindicator__item"></div>
- <div class="b-nav__contri navindicator__item"></div>
- <div class="b-nav__security navindicator__item"></div>
- </div>
- </nav>
- </div>
- <div class="mdcontent">
- <section>
- <div style="padding:10px 15px;">
- <div id="viewpage" name="viewpage">
- <div class="row">
- <div class="col-sm-12 col-md-12">
- <div>
-<h1>
-<a id="carbondata-pre-aggregate-datamap" class="anchor" href="#carbondata-pre-aggregate-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Pre-aggregate DataMap</h1>
-<ul>
-<li><a href="#quick-example">Quick Example</a></li>
-<li><a href="#datamap-management">DataMap Management</a></li>
-<li><a href="#preaggregate-datamap-introduction">Pre-aggregate Table</a></li>
-<li><a href="#loading-data">Loading Data</a></li>
-<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
-<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li>
-</ul>
-<h2>
-<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
-<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
-<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
-<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.2</pre></div>
-<p>Start spark-shell in new terminal, type :paste, then copy and run the following code.</p>
-<div class="highlight highlight-source-scala"><pre> <span class="pl-k">import</span> <span class="pl-en">java</span>.<span class="pl-en">io</span>.<span class="pl-en">File</span>
- <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.{<span class="pl-en">CarbonEnv</span>, <span class="pl-en">SparkSession</span>}
- <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">CarbonSession</span>.<span class="pl-en">_</span>
- <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">streaming</span>.{<span class="pl-en">ProcessingTime</span>, <span class="pl-en">StreamingQuery</span>}
- <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">carbondata</span>.<span class="pl-en">core</span>.<span class="pl-en">util</span>.<span class="pl-en">path</span>.<span class="pl-en">CarbonStorePath</span>
-
- <span class="pl-k">val</span> <span class="pl-smi">warehouse</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./warehouse<span class="pl-pds">"</span></span>).getCanonicalPath
- <span class="pl-k">val</span> <span class="pl-smi">metastore</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">File</span>(<span class="pl-s"><span class="pl-pds">"</span>./metastore<span class="pl-pds">"</span></span>).getCanonicalPath
-
- <span class="pl-k">val</span> <span class="pl-smi">spark</span> <span class="pl-k">=</span> <span class="pl-en">SparkSession</span>
- .builder()
- .master(<span class="pl-s"><span class="pl-pds">"</span>local<span class="pl-pds">"</span></span>)
- .appName(<span class="pl-s"><span class="pl-pds">"</span>preAggregateExample<span class="pl-pds">"</span></span>)
- .config(<span class="pl-s"><span class="pl-pds">"</span>spark.sql.warehouse.dir<span class="pl-pds">"</span></span>, warehouse)
- .getOrCreateCarbonSession(warehouse, metastore)
-
- spark.sparkContext.setLogLevel(<span class="pl-s"><span class="pl-pds">"</span>ERROR<span class="pl-pds">"</span></span>)
-
- <span class="pl-c"><span class="pl-c">//</span> drop table if exists previously</span>
- spark.sql(<span class="pl-k">s</span><span class="pl-s">"</span><span class="pl-s">DROP TABLE IF EXISTS sales</span><span class="pl-s">"</span>)
-
- <span class="pl-c"><span class="pl-c">//</span> Create main table</span>
- spark.sql(
- <span class="pl-k">s</span><span class="pl-s">"""</span>
-<span class="pl-s"> | CREATE TABLE sales (</span>
-<span class="pl-s"> | user_id string,</span>
-<span class="pl-s"> | country string,</span>
-<span class="pl-s"> | quantity int,</span>
-<span class="pl-s"> | price bigint)</span>
-<span class="pl-s"> | STORED AS carbondata</span>
-<span class="pl-s"> </span><span class="pl-s">"""</span>.stripMargin)
-
- <span class="pl-c"><span class="pl-c">//</span> Create pre-aggregate table on the main table</span>
- <span class="pl-c"><span class="pl-c">//</span> If main table already have data, following command </span>
- <span class="pl-c"><span class="pl-c">//</span> will trigger one immediate load to the pre-aggregate table</span>
- spark.sql(
- <span class="pl-k">s</span><span class="pl-s">"""</span>
-<span class="pl-s"> | CREATE DATAMAP agg_sales</span>
-<span class="pl-s"> | ON TABLE sales</span>
-<span class="pl-s"> | USING "preaggregate"</span>
-<span class="pl-s"> | AS</span>
-<span class="pl-s"> | SELECT country, sum(quantity), avg(price)</span>
-<span class="pl-s"> | FROM sales</span>
-<span class="pl-s"> | GROUP BY country</span>
-<span class="pl-s"> </span><span class="pl-s">"""</span>.stripMargin)
-
- <span class="pl-k">import</span> <span class="pl-en">spark</span>.<span class="pl-en">implicits</span>.<span class="pl-en">_</span>
- <span class="pl-k">import</span> <span class="pl-en">org</span>.<span class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span class="pl-en">sql</span>.<span class="pl-en">SaveMode</span>
- <span class="pl-k">import</span> <span class="pl-en">scala</span>.<span class="pl-en">util</span>.<span class="pl-en">Random</span>
-
- <span class="pl-c"><span class="pl-c">//</span> Load data to the main table, it will also</span>
- <span class="pl-c"><span class="pl-c">//</span> trigger immediate load to pre-aggregate table.</span>
- <span class="pl-c"><span class="pl-c">//</span> These two loading operation is carried out in a</span>
- <span class="pl-c"><span class="pl-c">//</span> transactional manner, meaning that the whole </span>
- <span class="pl-c"><span class="pl-c">//</span> operation will fail if one of the loading fails</span>
- <span class="pl-k">val</span> <span class="pl-smi">r</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-en">Random</span>()
- spark.sparkContext.parallelize(<span class="pl-c1">1</span> to <span class="pl-c1">10</span>)
- .map(x <span class="pl-k">=></span> (<span class="pl-s"><span class="pl-pds">"</span>ID.<span class="pl-pds">"</span></span> <span class="pl-k">+</span> r.nextInt(<span class="pl-c1">100000</span>), <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span> <span class="pl-k">+</span> x <span class="pl-k">%</span> <span class="pl-c1">8</span>, x <span class="pl-k">%</span> <span class="pl-c1">50</span>, x <span class="pl-k">%</span> <span class="pl- [...]
- .toDF(<span class="pl-s"><span class="pl-pds">"</span>user_id<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>country<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>price<span class="pl-pds">"</span></span>)
- .write
- .format(<span class="pl-s"><span class="pl-pds">"</span>carbondata<span class="pl-pds">"</span></span>)
- .option(<span class="pl-s"><span class="pl-pds">"</span>tableName<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>sales<span class="pl-pds">"</span></span>)
- .option(<span class="pl-s"><span class="pl-pds">"</span>compress<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>true<span class="pl-pds">"</span></span>)
- .mode(<span class="pl-en">SaveMode</span>.<span class="pl-en">Append</span>)
- .save()
-
- spark.sql(
- <span class="pl-k">s</span><span class="pl-s">"""</span>
-<span class="pl-s"> |SELECT country, sum(quantity), avg(price)</span>
-<span class="pl-s"> | from sales GROUP BY country</span>
-<span class="pl-s"> </span><span class="pl-s">"""</span>.stripMargin).show
-
- spark.stop</pre></div>
-<h4>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
-<p>DataMap can be created using following DDL</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-ON TABLE main_table
-USING "datamap_provider"
-DMPROPERTIES ('key'='value', ...)
-AS
- SELECT statement
-</code></pre>
-<p>The string followed by USING is called DataMap Provider, in this version CarbonData supports two
-kinds of DataMap:</p>
-<ol>
-<li>preaggregate, for pre-aggregate table. Pre-Aggregate table supports two values for DMPROPERTIES.
-a. 'path' is used to specify the store location of the datamap.('path'='/location/').
-b. 'partitioning' when set to false enables user to disable partitioning of the datamap.
-Default value is true for this property.</li>
-<li>timeseries, for timeseries roll-up table. Please refer to <a href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
-</li>
-</ol>
-<p>DataMap can be dropped using following DDL</p>
-<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
-ON TABLE main_table
-</code></pre>
-<p>To show all DataMaps created, use:</p>
-<pre><code>SHOW DATAMAP
-ON TABLE main_table
-</code></pre>
-<p>It will show all DataMaps created on main table.</p>
-<h2>
-<a id="preaggregate-datamap-introduction" class="anchor" href="#preaggregate-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Preaggregate DataMap Introduction</h2>
-<p>Pre-aggregate tables are created as DataMaps and managed as tables internally by CarbonData.
-User can create as many pre-aggregate datamaps required to improve query performance,
-provided the storage requirements and loading speeds are acceptable.</p>
-<p>Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer extension supports to
-select the most efficient pre-aggregate datamap and rewrite the SQL to query against the selected
-datamap instead of the main table. Since the data size of pre-aggregate datamap is smaller,
-user queries are much faster. In our previous experience, we have seen 5X to 100X times faster
-in production SQLs.</p>
-<p>For instance, main table called <strong>sales</strong> which is defined as</p>
-<pre><code>CREATE TABLE sales (
- order_time timestamp,
- user_id string,
- sex string,
- country string,
- quantity int,
- price bigint)
-STORED AS carbondata
-</code></pre>
-<p>User can create pre-aggregate tables using the Create DataMap DDL</p>
-<pre><code>CREATE DATAMAP agg_sales
-ON TABLE sales
-USING "preaggregate"
-AS
- SELECT country, sex, sum(quantity), avg(price)
- FROM sales
- GROUP BY country, sex
-</code></pre>
-<p><strong>NOTE</strong>:</p>
-<ul>
-<li>Preaggregate datamap is deprecated and it is replaced by MV datamap.
-Please refer <a href="./mv-datamap-guide.html">CarbonData MV DataMap</a> for more info.</li>
-</ul>
-<h4>
-<a id="functions-supported-in-pre-aggregate-table" class="anchor" href="#functions-supported-in-pre-aggregate-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Functions supported in pre-aggregate table</h4>
-<table>
-<thead>
-<tr>
-<th>Function</th>
-<th align="center">Rollup supported</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>SUM</td>
-<td align="center">Yes</td>
-</tr>
-<tr>
-<td>AVG</td>
-<td align="center">Yes</td>
-</tr>
-<tr>
-<td>MAX</td>
-<td align="center">Yes</td>
-</tr>
-<tr>
-<td>MIN</td>
-<td align="center">Yes</td>
-</tr>
-<tr>
-<td>COUNT</td>
-<td align="center">Yes</td>
-</tr>
-</tbody>
-</table>
-<h4>
-<a id="how-pre-aggregate-tables-are-selected" class="anchor" href="#how-pre-aggregate-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How pre-aggregate tables are selected</h4>
-<p>When a user query is submitted, during query planning phase, CarbonData will collect all matched
-pre-aggregate tables as candidates according to Relational Algebra transformation rules. Then, the
-best pre-aggregate table for this query will be selected among the candidates based on cost.
-For simplicity, current cost estimation is based on the data size of the pre-aggregate table. (We
-assume that query will be faster on smaller table)</p>
-<p>For the main table <strong>sales</strong> and pre-aggregate table <strong>agg_sales</strong> created above, following queries</p>
-<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
-
-SELECT sex, sum(quantity) from sales GROUP BY sex
-
-SELECT avg(price), country from sales GROUP BY country
-</code></pre>
-<p>will be transformed by CarbonData's query planner to query against pre-aggregate table
-<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
-<p>However, for following queries</p>
-<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
-
-SELECT sex, avg(quantity) from sales GROUP BY sex
-
-SELECT country, max(price) from sales GROUP BY country
-</code></pre>
-<p>will query against main table <strong>sales</strong> only, because it does not satisfy pre-aggregate table
-selection logic.</p>
-<h2>
-<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
-<p>For existing table with loaded data, data load to pre-aggregate table will be triggered by the
-CREATE DATAMAP statement when user creates the pre-aggregate table. For incremental loads after
-aggregates tables are created, loading data to main table triggers the load to pre-aggregate tables
-once main table loading is complete.</p>
-<p>These loads are transactional
-meaning that data on main table and pre-aggregate tables are only visible to the user after all
-tables are loaded successfully, if one of these loads fails, new data are not visible in all tables
-as if the load operation is not happened.</p>
-<h2>
-<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
-<p>As a technique for query acceleration, Pre-aggregate tables cannot be queried directly.
-Queries are to be made on main table. While doing query planning, internally CarbonData will check
-associated pre-aggregate tables with the main table, and do query plan transformation accordingly.</p>
-<p>User can verify whether a query can leverage pre-aggregate table or not by executing <code>EXPLAIN</code>
-command, which will show the transformed logical plan, and thus user can check whether pre-aggregate
-table is selected.</p>
-<h2>
-<a id="compacting-pre-aggregate-tables" class="anchor" href="#compacting-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting pre-aggregate tables</h2>
-<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table will <strong>not automatically</strong>
-compact the pre-aggregate tables created on the main table. User need to run Compaction command
-separately on each pre-aggregate table to compact them.</p>
-<p>Compaction is an optional operation for pre-aggregate table. If compaction is performed on
-main table but not performed on pre-aggregate table, all queries still can benefit from
-pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables
-can be triggered to merge the segments and files in the pre-aggregate tables.</p>
-<h2>
-<a id="data-management-with-pre-aggregate-tables" class="anchor" href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with pre-aggregate tables</h2>
-<p>In current implementation, data consistency needs to be maintained for both main table and pre-aggregate
-tables. Once there is pre-aggregate table created on the main table, following command on the main
-table
-is not supported:</p>
-<ol>
-<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
-<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
-<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported, and for dropping columns and
-change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
-not, the operation is allowed, otherwise operation will be rejected by throwing exception.</li>
-<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>
-</li>
-<li>Complex Datatypes for preaggregate is not supported.</li>
-</ol>
-<p>However, there is still way to support these operations on main table, in current CarbonData
-release, user can do as following:</p>
-<ol>
-<li>Remove the pre-aggregate table by <code>DROP DATAMAP</code> command</li>
-<li>Carry out the data management operation on main table</li>
-<li>Create the pre-aggregate table again by <code>CREATE DATAMAP</code> command
-Basically, user can manually trigger the operation by re-building the datamap.</li>
-</ol>
-<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
-
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
-</script></div>
-</div>
-</div>
-</div>
-<div class="doc-footer">
- <a href="#top" class="scroll-top">Top</a>
-</div>
-</div>
-</section>
-</div>
-</div>
-</div>
-</section><!-- End systemblock part -->
-<script src="js/custom.js"></script>
-</body>
-</html>
\ No newline at end of file
diff --git a/src/main/webapp/presto-guide.html b/content/prestodb-guide.html
similarity index 91%
copy from src/main/webapp/presto-guide.html
copy to content/prestodb-guide.html
index cb4469e..565b598 100644
--- a/src/main/webapp/presto-guide.html
+++ b/content/prestodb-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,7 +240,7 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="presto-guide" class="anchor" href="#presto-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Presto guide</h1>
+<a id="prestodb-guide" class="anchor" href="#prestodb-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prestodb guide</h1>
<p>This tutorial provides a quick introduction to using current integration/presto module.</p>
<p><a href="#presto-multinode-cluster-setup-for-carbondata">Presto Multinode Cluster Setup for Carbondata</a></p>
<p><a href="#presto-single-node-setup-for-carbondata">Presto Single Node Setup for Carbondata</a></p>
@@ -240,8 +249,8 @@
<h3>
<a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3>
<p>To know about which version of presto is supported by this version of carbon, visit
-<a href="https://github.com/apache/carbondata/blob/master/integration/presto/pom.xml" target=_blank>https://github.com/apache/carbondata/blob/master/integration/presto/pom.xml</a>
-and look for <code><presto.version></code></p>
+<a href="https://github.com/apache/carbondata/blob/master/pom.xml" target=_blank>https://github.com/apache/carbondata/blob/master/pom.xml</a>
+and look for <code><presto.version></code> inside <code>prestodb</code> profile.</p>
<p><em>Example:</em>
<code><presto.version>0.217</presto.version></code>
This means current version of carbon supports presto 0.217 version.</p>
@@ -344,12 +353,16 @@ discovery.uri=<coordinator_ip>:8086
<a id="configuring-carbondata-in-presto" class="anchor" href="#configuring-carbondata-in-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring Carbondata in Presto</h5>
<ol>
<li>Create a file named <code>carbondata.properties</code> in the <code>catalog</code> folder and set the required properties on all the nodes.</li>
+<li>As carbondata connector extends hive connector all the configurations(including S3) is same as hive connector.
+Just replace the connector name in hive configuration and copy same to carbondata.properties
+<code>connector.name = carbondata</code>
+</li>
</ol>
<h3>
<a id="add-plugins" class="anchor" href="#add-plugins" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Add Plugins</h3>
<ol>
<li>Create a directory named <code>carbondata</code> in plugin directory of presto.</li>
-<li>Copy <code>carbondata</code> jars to <code>plugin/carbondata</code> directory on all nodes.</li>
+<li>Copy all the jars from ../integration/presto/target/carbondata-presto-X.Y.Z-SNAPSHOT to <code>plugin/carbondata</code> directory on all nodes.</li>
</ol>
<h3>
<a id="start-presto-server-on-all-nodes" class="anchor" href="#start-presto-server-on-all-nodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto Server on all nodes</h3>
@@ -421,11 +434,12 @@ node.data-dir=/Users/apple/DEMO/presto_test/data
<p>Firstly: Compile carbondata, including carbondata-presto integration module</p>
<pre><code>$ git clone https://github.com/apache/carbondata
$ cd carbondata
-$ mvn -DskipTests -P{spark-version} -Dspark.version={spark-version-number} -Dhadoop.version={hadoop-version-number} clean package
+$ mvn -DskipTests -P{spark-version} -P{prestodb/prestosql} -Dspark.version={spark-version-number} -Dhadoop.version={hadoop-version-number} clean package
</code></pre>
<p>Replace the spark and hadoop version with the version used in your cluster.
-For example, if you are using Spark 2.2.1 and Hadoop 2.7.2, you would like to compile using:</p>
-<pre><code>mvn -DskipTests -Pspark-2.2 -Dspark.version=2.2.1 -Dhadoop.version=2.7.2 clean package
+For example, use prestodb profile and
+if you are using Spark 2.4.5, you would like to compile using:</p>
+<pre><code>mvn -DskipTests -Pspark-2.4 -Pprestodb -Dspark.version=2.4.5 -Dhadoop.version=2.7.2 clean package
</code></pre>
<p>Secondly: Create a folder named 'carbondata' under $PRESTO_HOME$/plugin and
copy all jars from carbondata/integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT
@@ -480,9 +494,9 @@ carbondata files.</p>
<h3>
<a id="supported-features-of-presto-carbon" class="anchor" href="#supported-features-of-presto-carbon" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Supported features of presto carbon</h3>
<p>Presto carbon only supports reading the carbon table which is written by spark carbon or carbon SDK.
-During reading, it supports the non-distributed datamaps like block datamap and bloom datamap.
-It doesn't support MV datamap and Pre-aggregate datamap as it needs query plan to be changed and presto does not allow it.
-Also Presto carbon supports streaming segment read from streaming table created by spark.</p>
+During reading, it supports the non-distributed indexes like block index and bloom index.
+It doesn't support Materialized View as it needs query plan to be changed and presto does not allow it.
+Also, Presto carbon supports streaming segment read from streaming table created by spark.</p>
<script>
// Show selected style on nav item
$(function() { $('.b-nav__quickstart').addClass('selected'); });
diff --git a/src/main/webapp/presto-guide.html b/content/prestosql-guide.html
similarity index 85%
copy from src/main/webapp/presto-guide.html
copy to content/prestosql-guide.html
index cb4469e..b4adeac 100644
--- a/src/main/webapp/presto-guide.html
+++ b/content/prestosql-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,7 +240,7 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="presto-guide" class="anchor" href="#presto-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Presto guide</h1>
+<a id="prestosql-guide" class="anchor" href="#prestosql-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prestosql guide</h1>
<p>This tutorial provides a quick introduction to using current integration/presto module.</p>
<p><a href="#presto-multinode-cluster-setup-for-carbondata">Presto Multinode Cluster Setup for Carbondata</a></p>
<p><a href="#presto-single-node-setup-for-carbondata">Presto Single Node Setup for Carbondata</a></p>
@@ -240,29 +249,29 @@
<h3>
<a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3>
<p>To know about which version of presto is supported by this version of carbon, visit
-<a href="https://github.com/apache/carbondata/blob/master/integration/presto/pom.xml" target=_blank>https://github.com/apache/carbondata/blob/master/integration/presto/pom.xml</a>
-and look for <code><presto.version></code></p>
+<a href="https://github.com/apache/carbondata/blob/master/pom.xml" target=_blank>https://github.com/apache/carbondata/blob/master/pom.xml</a>
+and look for <code><presto.version></code> inside <code>prestosql</code> profile.</p>
<p><em>Example:</em>
-<code><presto.version>0.217</presto.version></code>
-This means current version of carbon supports presto 0.217 version.</p>
+<code><presto.version>316</presto.version></code>
+This means current version of carbon supports presto 316 version.</p>
<p><em>Note:</em>
Currently carbondata supports only one version of presto, cannot handle multiple versions at same time. If user wish to use older version of presto, then need to use older version of carbon (other old branches, say branch-1.5 and check the supported presto version in it's pom.xml file in integration/presto/)</p>
<ol>
-<li>Download that version of Presto (say 0.217) using below command:</li>
+<li>Download that version of Presto (say 316) using below command:</li>
</ol>
-<pre><code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.217/presto-server-0.217.tar.gz
+<pre><code>wget https://repo1.maven.org/maven2/io/prestosql/presto-server/316/presto-server-316.tar.gz
</code></pre>
<ol start="2">
<li>
-<p>Extract Presto tar file: <code>tar zxvf presto-server-0.217.tar.gz</code>.</p>
+<p>Extract Presto tar file: <code>tar zxvf presto-server-316.tar.gz</code>.</p>
</li>
<li>
-<p>Download the Presto CLI of the same presto server version (say 0.217) for the coordinator and name it presto.</p>
+<p>Download the Presto CLI of the same presto server version (say 316) for the coordinator and name it presto.</p>
</li>
</ol>
-<pre><code> wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.217/presto-cli-0.217-executable.jar
+<pre><code> wget https://repo1.maven.org/maven2/io/prestosql/presto-cli/316/presto-cli-316-executable.jar
- mv presto-cli-0.217-executable.jar presto
+ mv presto-cli-316-executable.jar presto
chmod +x presto
</code></pre>
@@ -270,7 +279,7 @@ Currently carbondata supports only one version of presto, cannot handle multiple
<a id="create-configuration-files" class="anchor" href="#create-configuration-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Configuration Files</h3>
<ol>
<li>
-<p>Create <code>etc</code> folder in presto-server-0.217 directory.</p>
+<p>Create <code>etc</code> folder in presto-server-316 directory.</p>
</li>
<li>
<p>Create <code>config.properties</code>, <code>jvm.config</code>, <code>log.properties</code>, and <code>node.properties</code> files.</p>
@@ -302,7 +311,7 @@ node.data-dir=/home/ubuntu/data
</code></pre>
<h5>
<a id="contents-of-your-logproperties-file" class="anchor" href="#contents-of-your-logproperties-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your log.properties file</h5>
-<pre><code>com.facebook.presto=INFO
+<pre><code>io.prestosql=INFO
</code></pre>
<p>The default minimum level is <code>INFO</code>. There are four levels: <code>DEBUG</code>, <code>INFO</code>, <code>WARN</code> and <code>ERROR</code>.</p>
<h2>
@@ -344,19 +353,23 @@ discovery.uri=<coordinator_ip>:8086
<a id="configuring-carbondata-in-presto" class="anchor" href="#configuring-carbondata-in-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring Carbondata in Presto</h5>
<ol>
<li>Create a file named <code>carbondata.properties</code> in the <code>catalog</code> folder and set the required properties on all the nodes.</li>
+<li>As carbondata connector extends hive connector all the configurations(including S3) is same as hive connector.
+Just replace the connector name in hive configuration and copy same to carbondata.properties
+<code>connector.name = carbondata</code>
+</li>
</ol>
<h3>
<a id="add-plugins" class="anchor" href="#add-plugins" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Add Plugins</h3>
<ol>
<li>Create a directory named <code>carbondata</code> in plugin directory of presto.</li>
-<li>Copy <code>carbondata</code> jars to <code>plugin/carbondata</code> directory on all nodes.</li>
+<li>Copy all the jars from ../integration/presto/target/carbondata-presto-X.Y.Z-SNAPSHOT to <code>plugin/carbondata</code> directory on all nodes.</li>
</ol>
<h3>
<a id="start-presto-server-on-all-nodes" class="anchor" href="#start-presto-server-on-all-nodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto Server on all nodes</h3>
-<pre><code>./presto-server-0.217/bin/launcher start
+<pre><code>./presto-server-316/bin/launcher start
</code></pre>
<p>To run it as a background process.</p>
-<pre><code>./presto-server-0.217/bin/launcher run
+<pre><code>./presto-server-316/bin/launcher run
</code></pre>
<p>To run it in foreground.</p>
<h3>
@@ -373,9 +386,9 @@ discovery.uri=<coordinator_ip>:8086
<h3>
<a id="config-presto-server" class="anchor" href="#config-presto-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Config presto server</h3>
<ul>
-<li>Download presto server (0.217 is suggested and supported) : <a href="https://repo1.maven.org/maven2/com/facebook/presto/presto-server/" target=_blank rel="nofollow">https://repo1.maven.org/maven2/com/facebook/presto/presto-server/</a>
+<li>Download presto server (316 is suggested and supported) : <a href="https://repo1.maven.org/maven2/io/prestosql/presto-server/" target=_blank rel="nofollow">https://repo1.maven.org/maven2/io/prestosql/presto-server/</a>
</li>
-<li>Finish presto configuration following <a href="https://prestodb.io/docs/current/installation/deployment.html" target=_blank rel="nofollow">https://prestodb.io/docs/current/installation/deployment.html</a>.
+<li>Finish presto configuration following <a href="https://prestosql.io/docs/current/installation/deployment.html" target=_blank rel="nofollow">https://prestosql.io/docs/current/installation/deployment.html</a>.
A configuration example:</li>
</ul>
<p><strong>config.properties</strong></p>
@@ -407,8 +420,8 @@ optimizer.optimize-hash-generation = false
</code></pre>
<p><code>carbon.properties.filepath</code> property is used to set the carbon.properties file path and it is recommended to set otherwise some features may not work. Please check the above example.</p>
<p><strong>log.properties</strong></p>
-<pre><code>com.facebook.presto=DEBUG
-com.facebook.presto.server.PluginManager=DEBUG
+<pre><code>io.prestosql=DEBUG
+io.prestosql.server.PluginManager=DEBUG
</code></pre>
<p><strong>node.properties</strong></p>
<pre><code>node.environment=carbondata
@@ -421,11 +434,12 @@ node.data-dir=/Users/apple/DEMO/presto_test/data
<p>Firstly: Compile carbondata, including carbondata-presto integration module</p>
<pre><code>$ git clone https://github.com/apache/carbondata
$ cd carbondata
-$ mvn -DskipTests -P{spark-version} -Dspark.version={spark-version-number} -Dhadoop.version={hadoop-version-number} clean package
+$ mvn -DskipTests -P{spark-version} -P{prestodb/prestosql} -Dspark.version={spark-version-number} -Dhadoop.version={hadoop-version-number} clean package
</code></pre>
<p>Replace the spark and hadoop version with the version used in your cluster.
-For example, if you are using Spark 2.2.1 and Hadoop 2.7.2, you would like to compile using:</p>
-<pre><code>mvn -DskipTests -Pspark-2.2 -Dspark.version=2.2.1 -Dhadoop.version=2.7.2 clean package
+For example, use prestosql profile and<br>
+if you are using Spark 2.4.5, you would like to compile using:</p>
+<pre><code>mvn -DskipTests -Pspark-2.4 -Pprestosql -Dspark.version=2.4.5 -Dhadoop.version=2.7.2 clean package
</code></pre>
<p>Secondly: Create a folder named 'carbondata' under $PRESTO_HOME$/plugin and
copy all jars from carbondata/integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT
@@ -436,7 +450,7 @@ to $PRESTO_HOME$/plugin/carbondata</p>
hive.metastore.uri=thrift://<host>:<port>
</code></pre>
<p>Carbondata becomes one of the supported format of presto hive plugin, so the configurations and setup is similar to hive connector of presto.
-Please refer <a href="https://prestodb.io/docs/current/connector/hive.html" target=_blank rel="nofollow">https://prestodb.io/docs/current/connector/hive.html</a> for more details.</p>
+Please refer <a href="https://prestosql.io/docs/current/connector/hive.html" target=_blank rel="nofollow">https://prestosql.io/docs/current/connector/hive.html</a> for more details.</p>
<p><strong>Note</strong>: Since carbon can work only with hive metastore, it is necessary that spark also connects to same metastore db for creating tables and updating tables.
All the operations done on spark will be reflected in presto immediately.
It is mandatory to create Carbon tables from spark using CarbonData 1.5.2 or greater version since input/output formats are updated in carbon table properly from this version.</p>
@@ -456,7 +470,7 @@ It is mandatory to create Carbon tables from spark using CarbonData 1.5.2 or gre
hive.s3.endpoint={value}
</code></pre>
-<p>Please refer <a href="https://prestodb.io/docs/current/connector/hive.html" target=_blank rel="nofollow">https://prestodb.io/docs/current/connector/hive.html</a> for more details on S3 integration.</p>
+<p>Please refer <a href="https://prestosql.io/docs/current/connector/hive.html" target=_blank rel="nofollow">https://prestosql.io/docs/current/connector/hive.html</a> for more details on S3 integration.</p>
</li>
</ul>
<h3>
@@ -468,7 +482,7 @@ carbondata files.</p>
<a id="query-carbondata-in-cli-of-presto" class="anchor" href="#query-carbondata-in-cli-of-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query carbondata in CLI of presto</h3>
<ul>
<li>
-<p>Download presto cli client of version 0.217 : <a href="https://repo1.maven.org/maven2/com/facebook/presto/presto-cli" target=_blank rel="nofollow">https://repo1.maven.org/maven2/com/facebook/presto/presto-cli</a></p>
+<p>Download presto cli client of version 316 : <a href="https://repo1.maven.org/maven2/io/prestosql/presto-cli/" target=_blank rel="nofollow">https://repo1.maven.org/maven2/io/prestosql/presto-cli/</a></p>
</li>
<li>
<p>Start CLI:</p>
@@ -480,9 +494,9 @@ carbondata files.</p>
<h3>
<a id="supported-features-of-presto-carbon" class="anchor" href="#supported-features-of-presto-carbon" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Supported features of presto carbon</h3>
<p>Presto carbon only supports reading the carbon table which is written by spark carbon or carbon SDK.
-During reading, it supports the non-distributed datamaps like block datamap and bloom datamap.
-It doesn't support MV datamap and Pre-aggregate datamap as it needs query plan to be changed and presto does not allow it.
-Also Presto carbon supports streaming segment read from streaming table created by spark.</p>
+During reading, it supports the non-distributed index like block index and bloom index.
+It doesn't support Materialized View as it needs query plan to be changed and presto does not allow it.
+Also, Presto carbon supports streaming segment read from streaming table created by spark.</p>
<script>
// Show selected style on nav item
$(function() { $('.b-nav__quickstart').addClass('selected'); });
diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html
index 2fcc80a..80d151a 100644
--- a/content/quick-start-guide.html
+++ b/content/quick-start-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -232,12 +241,12 @@
<div>
<h1>
<a id="quick-start" class="anchor" href="#quick-start" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick Start</h1>
-<p>This tutorial provides a quick introduction to using CarbonData. To follow along with this guide, first download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/" target=_blank rel="nofollow">CarbonData website</a>.Alternatively it can be created following <a href="https://github.com/apache/carbondata/tree/master/build" target=_blank>Building CarbonData</a> steps.</p>
+<p>This tutorial provides a quick introduction to use CarbonData. To follow along with this guide, download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/" target=_blank rel="nofollow">CarbonData website</a>. Alternatively, it can be created following <a href="https://github.com/apache/carbondata/tree/master/build" target=_blank>Building CarbonData</a> steps.</p>
<h2>
<a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h2>
<ul>
<li>
-<p>CarbonData supports Spark versions upto 2.2.1.Please download Spark package from <a href="https://spark.apache.org/downloads.html" target=_blank rel="nofollow">Spark website</a></p>
+<p>CarbonData supports Spark versions up to 2.4. Please download Spark package from <a href="https://spark.apache.org/downloads.html" target=_blank rel="nofollow">Spark website</a></p>
</li>
<li>
<p>Create a sample.csv file using the following commands. The CSV file is required for loading data into CarbonData</p>
@@ -255,9 +264,10 @@ EOF
<a id="integration" class="anchor" href="#integration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration</h2>
<h3>
<a id="integration-with-execution-engines" class="anchor" href="#integration-with-execution-engines" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration with Execution Engines</h3>
-<p>CarbonData can be integrated with Spark,Presto and Hive execution engines. The below documentation guides on Installing and Configuring with these execution engines.</p>
+<p>CarbonData can be integrated with Spark, Presto, Flink and Hive execution engines. The below documentation guides on Installing and Configuring with these execution engines.</p>
<h4>
<a id="spark" class="anchor" href="#spark" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Spark</h4>
+<p><a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-sql-cli">Installing and Configuring CarbonData to run locally with Spark SQL CLI</a></p>
<p><a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Installing and Configuring CarbonData to run locally with Spark Shell</a></p>
<p><a href="#installing-and-configuring-carbondata-on-standalone-spark-cluster">Installing and Configuring CarbonData on Standalone Spark Cluster</a></p>
<p><a href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster">Installing and Configuring CarbonData on Spark on YARN Cluster</a></p>
@@ -280,14 +290,65 @@ EOF
<a id="alluxio" class="anchor" href="#alluxio" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Alluxio</h4>
<p><a href="./alluxio-guide.html">CarbonData supports read and write with Alluxio</a></p>
<h2>
+<a id="installing-and-configuring-carbondata-to-run-locally-with-spark-sql-cli" class="anchor" href="#installing-and-configuring-carbondata-to-run-locally-with-spark-sql-cli" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData to run locally with Spark SQL CLI</h2>
+<p>This will work with spark 2.3+ versions. In Spark SQL CLI, it uses CarbonExtensions to customize the SparkSession with CarbonData's parser, analyzer, optimizer and physical planning strategy rules in Spark.
+To enable CarbonExtensions, we need to add the following configuration.</p>
+<table>
+<thead>
+<tr>
+<th>Key</th>
+<th>Value</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>spark.sql.extensions</td>
+<td>org.apache.spark.sql.CarbonExtensions</td>
+</tr>
+</tbody>
+</table>
+<p>Start Spark SQL CLI by running the following command in the Spark directory:</p>
+<pre><code>./bin/spark-sql --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+</code></pre>
+<h6>
+<a id="creating-a-table" class="anchor" href="#creating-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h6>
+<pre><code>CREATE TABLE IF NOT EXISTS test_table (
+ id string,
+ name string,
+ city string,
+ age Int)
+STORED AS carbondata;
+</code></pre>
+<p><strong>NOTE</strong>: CarbonExtensions only support "STORED AS carbondata" and "USING carbondata"</p>
+<h6>
+<a id="loading-data-to-a-table" class="anchor" href="#loading-data-to-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h6>
+<pre><code>LOAD DATA INPATH '/local-path/sample.csv' INTO TABLE test_table;
+
+LOAD DATA INPATH 'hdfs://hdfs-path/sample.csv' INTO TABLE test_table;
+</code></pre>
+<pre><code>insert into table test_table select '1', 'name1', 'city1', 1;
+</code></pre>
+<p><strong>NOTE</strong>: Please provide the real file path of <code>sample.csv</code> for the above script.
+If you get "tablestatus.lock" issue, please refer to <a href="faq.html">FAQ</a></p>
+<h6>
+<a id="query-data-from-a-table" class="anchor" href="#query-data-from-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data from a Table</h6>
+<pre><code>SELECT * FROM test_table;
+</code></pre>
+<pre><code>SELECT city, avg(age), sum(age)
+FROM test_table
+GROUP BY city;
+</code></pre>
+<h2>
<a id="installing-and-configuring-carbondata-to-run-locally-with-spark-shell" class="anchor" href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData to run locally with Spark Shell</h2>
-<p>Apache Spark Shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. Please visit <a href="http://spark.apache.org/docs/latest/" target=_blank rel="nofollow">Apache Spark Documentation</a> for more details on Spark shell.</p>
+<p>Apache Spark Shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. Please visit <a href="http://spark.apache.org/docs/latest/" target=_blank rel="nofollow">Apache Spark Documentation</a> for more details on the Spark shell.</p>
<h4>
<a id="basics" class="anchor" href="#basics" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Basics</h4>
+<h6>
+<a id="option-1-using-carbonsession-deprecated-since-20" class="anchor" href="#option-1-using-carbonsession-deprecated-since-20" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Option 1: Using CarbonSession (deprecated since 2.0)</h6>
<p>Start Spark shell by running the following command in the Spark directory:</p>
<pre><code>./bin/spark-shell --jars <carbondata assembly jar path>
</code></pre>
-<p><strong>NOTE</strong>: Path where packaged release of CarbonData was downloaded or assembly jar will be available after <a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>building CarbonData</a> and can be copied from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code></p>
+<p><strong>NOTE</strong>: Path where packaged release of CarbonData was downloaded or assembly jar will be available after <a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>building CarbonData</a> and can be copied from <code>./assembly/target/scala-2.1x/apache-carbondata_xxx.jar</code></p>
<p>In this shell, SparkSession is readily available as <code>spark</code> and Spark context is readily available as <code>sc</code>.</p>
<p>In order to create a CarbonSession we will have to configure it explicitly in the following manner :</p>
<ul>
@@ -307,10 +368,30 @@ import org.apache.spark.sql.CarbonSession._
<code>SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<carbon_store_path>", "<local metastore path>")</code>.</li>
<li>Data storage location can be specified by <code><carbon_store_path></code>, like <code>/carbon/data/store</code>, <code>hdfs://localhost:9000/carbon/data/store</code> or <code>s3a://carbon/data/store</code>.</li>
</ul>
+<h6>
+<a id="option-2-using-sparksession-with-carbonextensions" class="anchor" href="#option-2-using-sparksession-with-carbonextensions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Option 2: Using SparkSession with CarbonExtensions</h6>
+<p>Start Spark shell by running the following command in the Spark directory:</p>
+<pre><code>./bin/spark-shell --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+</code></pre>
+<p><strong>NOTE</strong></p>
+<ul>
+<li>In this flow, we can use the built-in SparkSession <code>spark</code> instead of <code>carbon</code>.
+We also can create a new SparkSession instead of the built-in SparkSession <code>spark</code> if need.
+It need to add "org.apache.spark.sql.CarbonExtensions" into spark configuration "spark.sql.extensions".
+<pre><code>SparkSession newSpark = SparkSession
+ .builder()
+ .config(sc.getConf)
+ .enableHiveSupport
+ .config("spark.sql.extensions","org.apache.spark.sql.CarbonExtensions")
+ .getOrCreate()
+</code></pre>
+</li>
+<li>Data storage location can be specified by "spark.sql.warehouse.dir".</li>
+</ul>
<h4>
<a id="executing-queries" class="anchor" href="#executing-queries" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Executing Queries</h4>
<h6>
-<a id="creating-a-table" class="anchor" href="#creating-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h6>
+<a id="creating-a-table-1" class="anchor" href="#creating-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h6>
<pre><code>carbon.sql(
s"""
| CREATE TABLE IF NOT EXISTS test_table(
@@ -321,14 +402,48 @@ import org.apache.spark.sql.CarbonSession._
| STORED AS carbondata
""".stripMargin)
</code></pre>
+<p><strong>NOTE</strong>:
+The following table list all supported syntax:</p>
+<table>
+<thead>
+<tr>
+<th>create table</th>
+<th>SparkSession with CarbonExtensions</th>
+<th>CarbonSession</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>STORED AS carbondata</td>
+<td>yes</td>
+<td>yes</td>
+</tr>
+<tr>
+<td>USING carbondata</td>
+<td>yes</td>
+<td>yes</td>
+</tr>
+<tr>
+<td>STORED BY 'carbondata'</td>
+<td>no</td>
+<td>yes</td>
+</tr>
+<tr>
+<td>STORED BY 'org.apache.carbondata.format'</td>
+<td>no</td>
+<td>yes</td>
+</tr>
+</tbody>
+</table>
+<p>We suggest to use CarbonExtensions instead of CarbonSession.</p>
<h6>
-<a id="loading-data-to-a-table" class="anchor" href="#loading-data-to-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h6>
+<a id="loading-data-to-a-table-1" class="anchor" href="#loading-data-to-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h6>
<pre><code>carbon.sql("LOAD DATA INPATH '/path/to/sample.csv' INTO TABLE test_table")
</code></pre>
<p><strong>NOTE</strong>: Please provide the real file path of <code>sample.csv</code> for the above script.
If you get "tablestatus.lock" issue, please refer to <a href="faq.html">FAQ</a></p>
<h6>
-<a id="query-data-from-a-table" class="anchor" href="#query-data-from-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data from a Table</h6>
+<a id="query-data-from-a-table-1" class="anchor" href="#query-data-from-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data from a Table</h6>
<pre><code>carbon.sql("SELECT * FROM test_table").show()
carbon.sql(
@@ -351,10 +466,10 @@ carbon.sql(
<a id="procedure" class="anchor" href="#procedure" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Procedure</h3>
<ol>
<li>
-<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code>.</p>
+<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/apache-carbondata_xxx.jar</code>.</p>
</li>
<li>
-<p>Copy <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> to <code>$SPARK_HOME/carbonlib</code> folder.</p>
+<p>Copy <code>./assembly/target/scala-2.1x/apache-carbondata_xxx.jar</code> to <code>$SPARK_HOME/carbonlib</code> folder.</p>
<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exist inside <code>$SPARK_HOME</code> path.</p>
</li>
<li>
@@ -392,29 +507,6 @@ carbon.sql(
</tbody>
</table>
<ol start="7">
-<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code> file:</li>
-</ol>
-<table>
-<thead>
-<tr>
-<th>Property</th>
-<th>Required</th>
-<th>Description</th>
-<th>Example</th>
-<th>Remark</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>carbon.storelocation</td>
-<td>NO</td>
-<td>Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
-<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
-<td>Propose to set HDFS directory</td>
-</tr>
-</tbody>
-</table>
-<ol start="8">
<li>Verify the installation. For example:</li>
</ol>
<pre><code>./bin/spark-shell \
@@ -422,7 +514,11 @@ carbon.sql(
--total-executor-cores 2 \
--executor-memory 2G
</code></pre>
-<p><strong>NOTE</strong>: Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</p>
+<p><strong>NOTE</strong>:</p>
+<ul>
+<li>property "carbon.storelocation" is deprecated in carbondata 2.0 version. Only the users who used this property in previous versions can still use it in carbon 2.0 version.</li>
+<li>Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</li>
+</ul>
<h2>
<a id="installing-and-configuring-carbondata-on-spark-on-yarn-cluster" class="anchor" href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Spark on YARN Cluster</h2>
<p>This section provides the procedure to install CarbonData on "Spark on YARN" cluster.</p>
@@ -438,7 +534,7 @@ carbon.sql(
<p>The following steps are only for Driver Nodes. (Driver nodes are the one which starts the spark context.)</p>
<ol>
<li>
-<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> and copy to <code>$SPARK_HOME/carbonlib</code> folder.</p>
+<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md" target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/apache-carbondata_xxx.jar</code> and copy to <code>$SPARK_HOME/carbonlib</code> folder.</p>
<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exists inside <code>$SPARK_HOME</code> path.</p>
</li>
<li>
@@ -502,29 +598,6 @@ mv carbondata.tar.gz carbonlib/
</tbody>
</table>
<ol start="5">
-<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code>:</li>
-</ol>
-<table>
-<thead>
-<tr>
-<th>Property</th>
-<th>Required</th>
-<th>Description</th>
-<th>Example</th>
-<th>Default Value</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>carbon.storelocation</td>
-<td>NO</td>
-<td>Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td>
-<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td>
-<td>Propose to set HDFS directory</td>
-</tr>
-</tbody>
-</table>
-<ol start="6">
<li>Verify the installation.</li>
</ol>
<pre><code>./bin/spark-shell \
@@ -535,6 +608,7 @@ mv carbondata.tar.gz carbonlib/
</code></pre>
<p><strong>NOTE</strong>:</p>
<ul>
+<li>property "carbon.storelocation" is deprecated in carbondata 2.0 version. Only the users who used this property in previous versions can still use it in carbon 2.0 version.</li>
<li>Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</li>
<li>If use Spark + Hive 1.1.X, it needs to add carbondata assembly jar and carbondata-hive jar into parameter 'spark.sql.hive.metastore.jars' in spark-default.conf file.</li>
</ul>
@@ -546,7 +620,28 @@ mv carbondata.tar.gz carbonlib/
<p>b. Run the following command to start the CarbonData thrift server.</p>
<pre><code>./bin/spark-submit \
--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
-$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
+$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR
+</code></pre>
+<table>
+<thead>
+<tr>
+<th>Parameter</th>
+<th>Description</th>
+<th>Example</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>CARBON_ASSEMBLY_JAR</td>
+<td>CarbonData assembly jar name present in the <code>$SPARK_HOME/carbonlib/</code> folder.</td>
+<td>apache-carbondata-xx.jar</td>
+</tr>
+</tbody>
+</table>
+<p>c. Run the following command to work with S3 storage.</p>
+<pre><code>./bin/spark-submit \
+--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
+$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <access_key> <secret_key> <endpoint>
</code></pre>
<table>
<thead>
@@ -560,12 +655,22 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
<tr>
<td>CARBON_ASSEMBLY_JAR</td>
<td>CarbonData assembly jar name present in the <code>$SPARK_HOME/carbonlib/</code> folder.</td>
-<td>carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar</td>
+<td>apache-carbondata-xx.jar</td>
+</tr>
+<tr>
+<td>access_key</td>
+<td>Access key for S3 storage</td>
+<td></td>
</tr>
<tr>
-<td>carbon_store_path</td>
-<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path.</td>
-<td><code>hdfs://<host_name>:port/user/hive/warehouse/carbon.store</code></td>
+<td>secret_key</td>
+<td>Secret key for S3 storage</td>
+<td></td>
+</tr>
+<tr>
+<td>endpoint</td>
+<td>Endpoint for connecting to S3 storage</td>
+<td></td>
</tr>
</tbody>
</table>
@@ -573,7 +678,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
<pre><code>./bin/spark-submit \
--conf spark.sql.hive.thriftServer.singleSession=true \
--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
-$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
+$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR
</code></pre>
<p><strong>But</strong> in single-session mode, if one user changes the database from one connection, the database of the other connections will be changed too.</p>
<p><strong>Examples</strong></p>
@@ -582,8 +687,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
</ul>
<pre><code>./bin/spark-submit \
--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
-$SPARK_HOME/carbonlib/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar \
-hdfs://<host_name>:port/user/hive/warehouse/carbon.store
+$SPARK_HOME/carbonlib/apache-carbondata-xxx.jar
</code></pre>
<ul>
<li>Start with Fixed executors and resources.</li>
@@ -594,8 +698,7 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store
--driver-memory 20G \
--executor-memory 250G \
--executor-cores 32 \
-$SPARK_HOME/carbonlib/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar \
-hdfs://<host_name>:port/user/hive/warehouse/carbon.store
+$SPARK_HOME/carbonlib/apache-carbondata-xxx.jar
</code></pre>
<h3>
<a id="connecting-to-carbondata-thrift-server-using-beeline" class="anchor" href="#connecting-to-carbondata-thrift-server-using-beeline" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Connecting to CarbonData Thrift Server Using Beeline.</h3>
@@ -608,130 +711,14 @@ Example
</code></pre>
<h2>
<a id="installing-and-configuring-carbondata-on-presto" class="anchor" href="#installing-and-configuring-carbondata-on-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Presto</h2>
-<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto. User need to create CarbonData Table and load data into it
+<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto. User needs to create CarbonData Table and load data into it
either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a> or <a href="./csdk-guide.html">C++ SDK</a>.
-Once the table is created,it can be queried from Presto.</strong></p>
-<h3>
-<a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3>
-<ol>
-<li>
-<p>Download the 0.210 version of Presto using:
-<code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz</code></p>
-</li>
-<li>
-<p>Extract Presto tar file: <code>tar zxvf presto-server-0.210.tar.gz</code>.</p>
-</li>
-<li>
-<p>Download the Presto CLI for the coordinator and name it presto.</p>
-</li>
-</ol>
-<pre><code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.210/presto-cli-0.210-executable.jar
-
-mv presto-cli-0.210-executable.jar presto
-
-chmod +x presto
-</code></pre>
-<h3>
-<a id="create-configuration-files" class="anchor" href="#create-configuration-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Configuration Files</h3>
-<ol>
-<li>Create <code>etc</code> folder in presto-server-0.210 directory.</li>
-<li>Create <code>config.properties</code>, <code>jvm.config</code>, <code>log.properties</code>, and <code>node.properties</code> files.</li>
-<li>Install uuid to generate a node.id.</li>
-</ol>
-<pre><code>sudo apt-get install uuid
-
-uuid
-</code></pre>
-<h5>
-<a id="contents-of-your-nodeproperties-file" class="anchor" href="#contents-of-your-nodeproperties-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your node.properties file</h5>
-<pre><code>node.environment=production
-node.id=<generated uuid>
-node.data-dir=/home/ubuntu/data
-</code></pre>
-<h5>
-<a id="contents-of-your-jvmconfig-file" class="anchor" href="#contents-of-your-jvmconfig-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your jvm.config file</h5>
-<pre><code>-server
--Xmx16G
--XX:+UseG1GC
--XX:G1HeapRegionSize=32M
--XX:+UseGCOverheadLimit
--XX:+ExplicitGCInvokesConcurrent
--XX:+HeapDumpOnOutOfMemoryError
--XX:OnOutOfMemoryError=kill -9 %p
-</code></pre>
-<h5>
-<a id="contents-of-your-logproperties-file" class="anchor" href="#contents-of-your-logproperties-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your log.properties file</h5>
-<pre><code>com.facebook.presto=INFO
-</code></pre>
-<p>The default minimum level is <code>INFO</code>. There are four levels: <code>DEBUG</code>, <code>INFO</code>, <code>WARN</code> and <code>ERROR</code>.</p>
-<h3>
-<a id="coordinator-configurations" class="anchor" href="#coordinator-configurations" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Coordinator Configurations</h3>
-<h5>
-<a id="contents-of-your-configproperties" class="anchor" href="#contents-of-your-configproperties" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your config.properties</h5>
-<pre><code>coordinator=true
-node-scheduler.include-coordinator=false
-http-server.http.port=8086
-query.max-memory=5GB
-query.max-total-memory-per-node=5GB
-query.max-memory-per-node=3GB
-memory.heap-headroom-per-node=1GB
-discovery-server.enabled=true
-discovery.uri=http://localhost:8086
-task.max-worker-threads=4
-optimizer.dictionary-aggregation=true
-optimizer.optimize-hash-generation = false
-</code></pre>
-<p>The options <code>node-scheduler.include-coordinator=false</code> and <code>coordinator=true</code> indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers.</p>
-<p><strong>Note</strong>: It is recommended to set <code>query.max-memory-per-node</code> to half of the JVM config max memory, though the workload is highly concurrent, lower value for <code>query.max-memory-per-node</code> is to be used.</p>
-<p>Also relation between below two configuration-properties should be like:
-If, <code>query.max-memory-per-node=30GB</code>
-Then, <code>query.max-memory=<30GB * number of nodes></code>.</p>
-<h3>
-<a id="worker-configurations" class="anchor" href="#worker-configurations" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Worker Configurations</h3>
-<h5>
-<a id="contents-of-your-configproperties-1" class="anchor" href="#contents-of-your-configproperties-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your config.properties</h5>
-<pre><code>coordinator=false
-http-server.http.port=8086
-query.max-memory=5GB
-query.max-memory-per-node=2GB
-discovery.uri=<coordinator_ip>:8086
-</code></pre>
-<p><strong>Note</strong>: <code>jvm.config</code> and <code>node.properties</code> files are same for all the nodes (worker + coordinator). All the nodes should have different <code>node.id</code>.(generated by uuid command).</p>
-<h3>
-<a id="catalog-configurations" class="anchor" href="#catalog-configurations" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Catalog Configurations</h3>
-<ol>
-<li>Create a folder named <code>catalog</code> in etc directory of presto on all the nodes of the cluster including the coordinator.</li>
-</ol>
-<h5>
-<a id="configuring-carbondata-in-presto" class="anchor" href="#configuring-carbondata-in-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring Carbondata in Presto</h5>
-<ol>
-<li>Create a file named <code>carbondata.properties</code> in the <code>catalog</code> folder and set the required properties on all the nodes.</li>
-</ol>
-<h3>
-<a id="add-plugins" class="anchor" href="#add-plugins" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Add Plugins</h3>
-<ol>
-<li>Create a directory named <code>carbondata</code> in plugin directory of presto.</li>
-<li>Copy <code>carbondata</code> jars to <code>plugin/carbondata</code> directory on all nodes.</li>
-</ol>
-<h3>
-<a id="start-presto-server-on-all-nodes" class="anchor" href="#start-presto-server-on-all-nodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto Server on all nodes</h3>
-<pre><code>./presto-server-0.210/bin/launcher start
-</code></pre>
-<p>To run it as a background process.</p>
-<pre><code>./presto-server-0.210/bin/launcher run
-</code></pre>
-<p>To run it in foreground.</p>
-<h3>
-<a id="start-presto-cli" class="anchor" href="#start-presto-cli" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto CLI</h3>
-<pre><code>./presto
-</code></pre>
-<p>To connect to carbondata catalog use the following command:</p>
-<pre><code>./presto --server <coordinator_ip>:8086 --catalog carbondata --schema <schema_name>
-</code></pre>
-<p>Execute the following command to ensure the workers are connected.</p>
-<pre><code>select * from system.runtime.nodes;
-</code></pre>
-<p>Now you can use the Presto CLI on the coordinator to query data sources in the catalog using the Presto workers.</p>
+Once the table is created, it can be queried from Presto.</strong></p>
+<p>Please refer the presto guide linked below.</p>
+<p>prestodb guide - <a href="./prestodb-guide.html">prestodb</a></p>
+<p>prestosql guide - <a href="./prestosql-guide.html">prestosql</a></p>
+<p>Once installed the presto with carbonData as per the above guide,
+you can use the Presto CLI on the coordinator to query data sources in the catalog using the Presto workers.</p>
<p>List the schemas(databases) available</p>
<pre><code>show schemas;
</code></pre>
@@ -744,8 +731,7 @@ discovery.uri=<coordinator_ip>:8086
<p>Query from the available tables</p>
<pre><code>select * from carbon_table;
</code></pre>
-<p><strong>Note :</strong> Create Tables and data loads should be done before executing queries as we can not create carbon table from this interface.</p>
-<pre><code></code></pre>
+<p><strong>Note:</strong> Create Tables and data loads should be done before executing queries as we can not create carbon table from this interface.</p>
<script>
// Show selected style on nav item
$(function() { $('.b-nav__quickstart').addClass('selected'); });
diff --git a/content/release-guide.html b/content/release-guide.html
index 42663e8..dffc15a 100644
--- a/content/release-guide.html
+++ b/content/release-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
diff --git a/content/s3-guide.html b/content/s3-guide.html
index df111ce..b086939 100644
--- a/content/s3-guide.html
+++ b/content/s3-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
diff --git a/src/main/webapp/s3-guide.html b/content/scd-and-cdc-guide.html
similarity index 73%
copy from src/main/webapp/s3-guide.html
copy to content/scd-and-cdc-guide.html
index df111ce..4e85573 100644
--- a/src/main/webapp/s3-guide.html
+++ b/content/scd-and-cdc-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,75 +240,61 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="s3-guide" class="anchor" href="#s3-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Guide</h1>
-<p>Object storage is the recommended storage format in cloud as it can support storing large data
-files. S3 APIs are widely used for accessing object stores. This can be
-used to store or retrieve data on Amazon cloud, Huawei Cloud(OBS) or on any other object
-stores conforming to S3 API.
-Storing data in cloud is advantageous as there are no restrictions on the size of
-data and the data can be accessed from anywhere at any time.
-Carbondata can support any Object Storage that conforms to Amazon S3 API.
-Carbondata relies on Hadoop provided S3 filesystem APIs to access Object stores.</p>
-<h1>
-<a id="writing-to-object-storage" class="anchor" href="#writing-to-object-storage" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writing to Object Storage</h1>
-<p>To store carbondata files onto Object Store, <code>carbon.storelocation</code> property will have
-to be configured with Object Store path in CarbonProperties file.</p>
-<p>For example:</p>
-<pre><code>carbon.storelocation=s3a://mybucket/carbonstore
-</code></pre>
-<p>If the existing store location cannot be changed or only specific tables need to be stored
-onto cloud object store, it can be done so by specifying the <code>location</code> option in the create
-table DDL command.</p>
-<p>For example:</p>
-<pre><code>CREATE TABLE IF NOT EXISTS db1.table1(col1 string, col2 int) STORED AS carbondata LOCATION 's3a://mybucket/carbonstore'
+<a id="upsert-into-a-carbon-dataset-using-merge" class="anchor" href="#upsert-into-a-carbon-dataset-using-merge" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Upsert into a Carbon DataSet using Merge</h1>
+<h2>
+<a id="scd-and-cdc-scenarios" class="anchor" href="#scd-and-cdc-scenarios" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SCD and CDC Scenarios</h2>
+<p>Change Data Capture (CDC), is to apply all data changes generated from an external data set
+into a target dataset. In other words, a set of changes (update/delete/insert) applied to an external
+table needs to be applied to a target table.</p>
+<p>Slowly Changing Dimensions (SCD), are the dimensions in which the data changes slowly, rather
+than changing regularly on a time basis.</p>
+<p>SCD and CDC data changes can be merged to a carbon dataset online using the data frame level <code>MERGE</code> API.</p>
+<h4>
+<a id="merge-api" class="anchor" href="#merge-api" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MERGE API</h4>
+<p>Below API merges the datasets online and applies the actions as per the conditions.</p>
+<pre><code> targetDS.merge(sourceDS, <condition>)
+ .whenMatched(<condition>)
+ .updateExpr(updateMap)
+ .insertExpr(insertMap_u)
+ .whenNotMatched(<condition>)
+ .insertExpr(insertMap)
+ .whenNotMatchedAndExistsOnlyOnTarget(<condition>)
+ .delete()
+ .execute()
</code></pre>
-<p>For more details on create table, Refer <a href="ddl-of-carbondata.html#create-table">DDL of CarbonData</a></p>
-<h1>
-<a id="authentication" class="anchor" href="#authentication" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Authentication</h1>
-<p>Authentication properties will have to be configured to store the carbondata files on to S3 location.</p>
-<p>Authentication properties can be set in any of the following ways:</p>
-<ol>
+<h4>
+<a id="merge-api-operation-semantics" class="anchor" href="#merge-api-operation-semantics" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MERGE API Operation Semantics</h4>
+<p>Below is the detailed description of the <code>merge</code> API operation.</p>
+<ul>
+<li>
+<code>merge</code> will merge the datasets based on a condition.</li>
+<li>
+<code>whenMatched</code> clauses are executed when a source row matches a target table row based on the match condition.
+These clauses have the following semantics.
+<ul>
<li>
-<p>Set authentication properties in core-site.xml, refer
-<a href="https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authentication_properties" rel="nofollow">hadoop authentication document</a></p>
+<code>whenMatched</code> clauses can have at most one updateExpr and one delete action. The <code>updateExpr</code> action in merge only updates the specified columns of the matched target row. The <code>delete</code> action deletes the matched row.</li>
+<li>If there are two <code>whenMatched</code> clauses, then they are evaluated in order they are specified. The first clause must have a clause condition (otherwise the second clause is never executed).</li>
+<li>If both <code>whenMatched</code> clauses have conditions and neither of the conditions are true for a matching source-target row pair, then the matched target row is left unchanged.</li>
+</ul>
</li>
<li>
-<p>Set authentication properties in spark-defaults.conf.</p>
+<code>whenNotMatched</code> clause is executed when a source row does not match any target row based on the match condition.
+<ul>
+<li>
+<code>whenNotMatched</code> clause can have only the <code>insertExpr</code> action. The new row is generated based on the specified column and corresponding expressions. Users do not need to specify all the columns in the target table. For unspecified target columns, NULL is inserted.</li>
+</ul>
</li>
-</ol>
-<p>Example</p>
-<pre><code>spark.hadoop.fs.s3a.secret.key=123
-spark.hadoop.fs.s3a.access.key=456
-</code></pre>
-<ol start="3">
-<li>Pass authentication properties with spark-submit as configuration.</li>
-</ol>
-<p>Example:</p>
-<pre><code>./bin/spark-submit \
---master yarn \
---conf spark.hadoop.fs.s3a.secret.key=123 \
---conf spark.hadoop.fs.s3a.access.key=456 \
---class=xxx
-</code></pre>
-<ol start="4">
-<li>Set authentication properties to hadoop configuration object in sparkContext.</li>
-</ol>
-<p>Example:</p>
-<pre><code>sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", "123")
-sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.access.key","456")
-</code></pre>
-<h1>
-<a id="recommendations" class="anchor" href="#recommendations" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Recommendations</h1>
-<ol>
-<li>Object Storage like S3 does not support file leasing mechanism(supported by HDFS) that is
-required to take locks which ensure consistency between concurrent operations therefore, it is
-recommended to set the configurable lock path property(<a href="./configuration-parameters.html#system-configuration">carbon.lock.path</a>)
-to a HDFS directory.</li>
-<li>Concurrent data manipulation operations are not supported. Object stores follow eventual consistency semantics, i.e., any put request might take some time to reflect when trying to list. This behaviour causes the data read is always not consistent or not the latest.</li>
-</ol>
+<li>
+<code>whenNotMatchedAndExistsOnlyOnTarget</code> clause is executed when row does not match source and exists only in target. This clause can have only delete action.</li>
+</ul>
+<p><strong>NOTE:</strong> SQL syntax for merge is not yet supported.</p>
+<h5>
+<a id="example-code-to-implement-cdcscd-scenario" class="anchor" href="#example-code-to-implement-cdcscd-scenario" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example code to implement cdc/scd scenario</h5>
+<p>Please refer example class <a href="https://github.com/apache/carbondata/blob/master/integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala" target=_blank>MergeTestCase</a> to understand and implement scd and cdc scenarios.</p>
<script>
// Show selected style on nav item
-$(function() { $('.b-nav__s3').addClass('selected'); });
+$(function() { $('.b-nav__quickstart').addClass('selected'); });
</script></div>
</div>
</div>
diff --git a/content/sdk-guide.html b/content/sdk-guide.html
index 1a356b6..c853984 100644
--- a/content/sdk-guide.html
+++ b/content/sdk-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -239,8 +248,8 @@
</ol>
<h1>
<a id="sdk-writer" class="anchor" href="#sdk-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Writer</h1>
-<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader.
-If user want to use SDK, except carbondata-store-sdk-x.x.x-SNAPSHOT.jar,
+<p>In the carbon jars package, there exist a carbondata-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader.
+If user want to use SDK, except carbondata-sdk-x.x.x-SNAPSHOT.jar,
it needs carbondata-core-x.x.x-SNAPSHOT.jar, carbondata-common-x.x.x-SNAPSHOT.jar,
carbondata-format-x.x.x-SNAPSHOT.jar, carbondata-hadoop-x.x.x-SNAPSHOT.jar and carbondata-processing-x.x.x-SNAPSHOT.jar.
What's more, user also can use carbondata-sdk.jar directly.</p>
@@ -258,7 +267,7 @@ These SDK writer output contains just carbondata and carbonindex files. No metad
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.util.CarbonProperties</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriterBuilder</span>;
-<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>;
+<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.Field</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Schema</span>;
<span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdk</span> {
@@ -302,7 +311,7 @@ These SDK writer output contains just carbondata and carbonindex files. No metad
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.DataTypes</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.AvroCarbonWriter</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>;
-<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>;
+<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.Field</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.avro.generic.GenericData</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.commons.lang.CharEncoding</span>;
@@ -358,7 +367,7 @@ These SDK writer output contains just carbondata and carbonindex files. No metad
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.util.CarbonProperties</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriterBuilder</span>;
-<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>;
+<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.Field</span>;
<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Schema</span>;
<span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdkJson</span> {
@@ -554,7 +563,7 @@ or directly use DataTypes.VARCHAR if it is carbon schema.</p>
<a id="example" class="anchor" href="#example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example</h3>
<pre><code>SELECT * FROM carbonfile.`$Path`
</code></pre>
-<p>Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/DirectSQLExample.scala" target=_blank>DirectSQLExample</a> in the CarbonData repo.</p>
+<p>Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/scala/org/apache/carbondata/examples/DirectSQLExample.scala" target=_blank>DirectSQLExample</a> in the CarbonData repo.</p>
<h2>
<a id="api-list" class="anchor" href="#api-list" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2>
<h3>
@@ -611,8 +620,7 @@ public CarbonWriterBuilder localDictionaryThreshold(int localDictionaryThreshold
* Sets the list of columns that needs to be in sorted order
*
* @param sortColumns is a string array of columns that needs to be sorted.
- * If it is null or by default all dimensions are selected for sorting
- * If it is empty array, no columns are sorted
+ * If it is null or empty array, no columns are selected for sorting.
* @return updated CarbonWriterBuilder
*/
public CarbonWriterBuilder sortBy(String[] sortColumns);
@@ -667,7 +675,7 @@ public CarbonWriterBuilder withLoadOptions(Map<String, String> options);
* c. local_dictionary_threshold -- positive value, default is 10000
* d. local_dictionary_enable -- true / false. Default is false
* e. sort_columns -- comma separated column. "c1,c2". Default no columns are sorted.
- * j. sort_scope -- "local_sort", "no_sort", "batch_sort". default value is "no_sort"
+ * j. sort_scope -- "local_sort", "no_sort". default value is "no_sort"
* k. long_string_columns -- comma separated string columns which are more than 32k length.
* default value is null.
* l. inverted_index -- comma separated string columns for which inverted index needs to be
@@ -791,7 +799,7 @@ public abstract void write(Object object) throws IOException;
public abstract void close() throws IOException;
</code></pre>
<h3>
-<a id="class-orgapachecarbondatasdkfilefield" class="anchor" href="#class-orgapachecarbondatasdkfilefield" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Field</h3>
+<a id="class-orgapachecarbondatacoremetadatadatatypefield" class="anchor" href="#class-orgapachecarbondatacoremetadatadatatypefield" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.core.metadata.datatype.Field</h3>
<pre><code>/**
* Field Constructor
*
@@ -867,9 +875,9 @@ while (reader.hasNext()) {
// 3. Close this reader
reader.close();
</code></pre>
-<p>Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java" target=_blank>CarbonReaderExample</a> in the CarbonData repo.</p>
+<p>Find example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java" target=_blank>CarbonReaderExample</a> in the CarbonData repo.</p>
<p>SDK reader also supports reading carbondata files and filling it to apache arrow vectors.
-Find example code at <a href="https://github.com/apache/carbondata/blob/master/store/sdk/src/test/java/org/apache/carbondata/sdk/file/ArrowCarbonReaderTest.java" target=_blank>ArrowCarbonReaderTest</a> in the CarbonData repo.</p>
+Find example code at <a href="https://github.com/apache/carbondata/blob/master/sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/ArrowCarbonReaderTest.java" target=_blank>ArrowCarbonReaderTest</a> in the CarbonData repo.</p>
<h2>
<a id="api-list-1" class="anchor" href="#api-list-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2>
<h3>
@@ -1183,7 +1191,7 @@ public static Schema parseJson(String json);
public Schema asOriginOrder();
</code></pre>
<h3>
-<a id="class-orgapachecarbondatasdkfilefield-1" class="anchor" href="#class-orgapachecarbondatasdkfilefield-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Field</h3>
+<a id="class-orgapachecarbondatacoremetadatadatatypefield-1" class="anchor" href="#class-orgapachecarbondatacoremetadatadatatypefield-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.core.metadata.datatype.Field</h3>
<pre><code>/**
* Field Constructor
*
@@ -1199,7 +1207,7 @@ public Field(String name, String type);
*/
public Field(ColumnSchema columnSchema);
</code></pre>
-<p>Find S3 example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java" target=_blank>SDKS3Example</a> in the CarbonData repo.</p>
+<p>Find S3 example code at <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java" target=_blank>SDKS3Example</a> in the CarbonData repo.</p>
<h1>
<a id="common-api-list-for-carbonreader-and-carbonwriter" class="anchor" href="#common-api-list-for-carbonreader-and-carbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Common API List for CarbonReader and CarbonWriter</h1>
<h3>
diff --git a/src/main/webapp/mv-datamap-guide.html b/content/secondary-index-guide.html
similarity index 60%
rename from src/main/webapp/mv-datamap-guide.html
rename to content/secondary-index-guide.html
index 22cc201..da5c71e 100644
--- a/src/main/webapp/mv-datamap-guide.html
+++ b/content/secondary-index-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,34 +240,47 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-mv-datamap" class="anchor" href="#carbondata-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData MV DataMap</h1>
+<a id="carbondata-secondary-index" class="anchor" href="#carbondata-secondary-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Secondary Index</h1>
<ul>
<li><a href="#quick-example">Quick Example</a></li>
-<li><a href="#mv-datamap-introduction">MV DataMap</a></li>
+<li><a href="#Secondary-Index-Introduction">Secondary Index Table</a></li>
<li><a href="#loading-data">Loading Data</a></li>
<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#compacting-mv-datamap">Compaction</a></li>
-<li><a href="#data-management-with-mv-tables">Data Management</a></li>
+<li><a href="#compacting-SI-table">Compaction</a></li>
+<li><a href="#DDLs-on-Secondary-Index">DDLs on Secondary Index</a></li>
</ul>
<h2>
<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
<p>Start spark-sql in terminal and run the following queries,</p>
-<pre><code>CREATE TABLE maintable(a int, b string, c int) stored by 'carbondata';
-insert into maintable select 1, 'ab', 2;
-CREATE DATAMAP datamap_1 on table maintable as SELECT a, sum(b) from maintable group by a;
-SELECT a, sum(b) from maintable group by a;
-// NOTE: run explain query and check if query hits the datamap table from the plan
-EXPLAIN SELECT a, sum(b) from maintable group by a;
+<pre><code>CREATE TABLE maintable(a int, b string, c string) stored as carbondata;
+insert into maintable select 1, 'ab', 'cd';
+CREATE index index1 on table maintable(c) AS 'carbondata';
+SELECT a from maintable where c = 'cd';
+// NOTE: run explain query and check if query hits the SI table from the plan
+EXPLAIN SELECT a from maintable where c = 'cd';
</code></pre>
<h2>
-<a id="mv-datamap-introduction" class="anchor" href="#mv-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MV DataMap Introduction</h2>
-<p>MV tables are created as DataMaps and managed as tables internally by CarbonData. User can create
-limitless MV datamaps on a table to improve query performance provided the storage requirements
-and loading time is acceptable.</p>
-<p>MV datamap can be a lazy or a non-lazy datamap. Once MV datamaps are created, CarbonData's
-CarbonAnalyzer helps to select the most efficient MV datamap based on the user query and rewrite
-the SQL to select the data from MV datamap instead of main table. Since the data size of MV
-datamap is smaller and data is pre-processed, user queries are much faster.</p>
+<a id="secondary-index-introduction" class="anchor" href="#secondary-index-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Secondary Index Introduction</h2>
+<p>Secondary index tables are created as indexes and managed as child tables internally by
+Carbondata. Users can create a secondary index based on the column position in the main table(Recommended
+for right columns) and the queries should have filter on that column to improve the filter query
+performance.</p>
+<p>Data refresh to the secondary index is always automatic. Once SI table is created, Carbondata's
+CarbonOptimizer with the help of <code>CarbonSITransformationRule</code>, transforms the query plan to hit the
+SI table based on the filter condition or set of filter conditions present in the query.
+So the first level of pruning will be done on the SI table as it stores blocklets and main table/parent
+table pruning will be based on the SI output, which helps in giving the faster query results with
+better pruning.</p>
+<p>Secondary Index table can be created with the below syntax</p>
+<pre><code>CREATE INDEX [IF NOT EXISTS] index_name
+ON TABLE maintable(index_column)
+AS
+'carbondata'
+[PROPERTIES('table_blocksize'='1')]
+</code></pre>
+<blockquote>
+<p>NOTE: Keywords given inside <code>[]</code> is optional.</p>
+</blockquote>
<p>For instance, main table called <strong>sales</strong> which is defined as</p>
<pre><code>CREATE TABLE sales (
order_time timestamp,
@@ -269,168 +291,103 @@ datamap is smaller and data is pre-processed, user queries are much faster.</p>
price bigint)
STORED AS carbondata
</code></pre>
-<p>User can create MV tables using the Create DataMap DDL</p>
-<pre><code>CREATE DATAMAP agg_sales
-ON TABLE sales
-USING "MV"
-DMPROPERTIES('TABLE_BLOCKSIZE'='256 MB','LOCAL_DICTIONARY_ENABLE'='false')
+<p>User can create SI table using the Create Index DDL</p>
+<pre><code>CREATE INDEX index_sales
+ON TABLE sales(user_id)
AS
- SELECT country, sex, sum(quantity), avg(price)
- FROM sales
- GROUP BY country, sex
+'carbondata'
+PROPERTIES('table_blocksize'='1')
</code></pre>
-<p><strong>NOTE</strong>:</p>
-<ul>
-<li>
-<p>Group by/Filter columns has to be provided in projection list while creating mv datamap</p>
-</li>
-<li>
-<p>If only single parent table is involved in mv datamap creation, then TableProperties of Parent table
-(if not present in a aggregate function like sum(col)) listed below will be
-inherited to datamap table</p>
-<ol>
-<li>SORT_COLUMNS</li>
-<li>SORT_SCOPE</li>
-<li>TABLE_BLOCKSIZE</li>
-<li>FLAT_FOLDER</li>
-<li>LONG_STRING_COLUMNS</li>
-<li>LOCAL_DICTIONARY_ENABLE</li>
-<li>LOCAL_DICTIONARY_THRESHOLD</li>
-<li>LOCAL_DICTIONARY_EXCLUDE</li>
-<li>DICTIONARY_INCLUDE</li>
-<li>DICTIONARY_EXCLUDE</li>
-<li>INVERTED_INDEX</li>
-<li>NO_INVERTED_INDEX</li>
-<li>COLUMN_COMPRESSOR</li>
-</ol>
-</li>
-<li>
-<p>All columns of main table at once cannot participate in mv datamap table creation</p>
-</li>
-<li>
-<p>TableProperties can be provided in DMProperties excluding LOCAL_DICTIONARY_INCLUDE,
-LOCAL_DICTIONARY_EXCLUDE, DICTIONARY_INCLUDE, DICTIONARY_EXCLUDE, INVERTED_INDEX,
-NO_INVERTED_INDEX, SORT_COLUMNS, LONG_STRING_COLUMNS, RANGE_COLUMN & COLUMN_META_CACHE</p>
-</li>
-<li>
-<p>TableProperty given in DMProperties will be considered for mv creation, eventhough if same
-property is inherited from parent table, which allows user to provide different tableproperties
-for child table</p>
-</li>
-<li>
-<p>MV creation with limit or union all ctas queries is unsupported</p>
-</li>
-<li>
-<p>MV datamap does not support Streaming</p>
-</li>
-</ul>
<h4>
-<a id="how-mv-tables-are-selected" class="anchor" href="#how-mv-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How MV tables are selected</h4>
-<p>When a user query is submitted, during query planning phase, CarbonData will collect modular plan
-candidates and process the the ModularPlan based on registered summary data sets. Then,
-mv datamap table for this query will be selected among the candidates.</p>
-<p>For the main table <strong>sales</strong> and mv table <strong>agg_sales</strong> created above, following queries</p>
-<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+<a id="how-si-tables-are-selected" class="anchor" href="#how-si-tables-are-selected" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How SI tables are selected</h4>
+<p>When a user executes a filter query, during the query planning phase, CarbonData with the help of
+<code>CarbonSITransformationRule</code>, checks if there are any index tables present on the filter column of
+query. If there are any, then the filter query plan will be transformed in such a way that execution will
+first hit the corresponding SI table and give input to the main table for further pruning.</p>
+<p>For the main table <strong>sales</strong> and SI table <strong>index_sales</strong> created above, following queries</p>
+<pre><code>SELECT country, sex from sales where user_id = 'xxx'
-SELECT sex, sum(quantity) from sales GROUP BY sex
-
-SELECT avg(price), country from sales GROUP BY country
-</code></pre>
-<p>will be transformed by CarbonData's query planner to query against mv table
-<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
-<p>However, for following queries</p>
-<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY user_id, country, sex
-
-SELECT sex, avg(quantity) from sales GROUP BY sex
-
-SELECT country, max(price) from sales GROUP BY country
+SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA'
</code></pre>
-<p>will query against main table <strong>sales</strong> only, because it does not satisfy mv table
-selection logic.</p>
+<p>will be transformed by CarbonData's <code>CarbonSITransformationRule</code> to query against SI table
+<strong>index_sales</strong> first which will be input to the main table <strong>sales</strong></p>
<h2>
<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data</h2>
<h3>
-<a id="loading-data-to-non-lazy-mv-datamap" class="anchor" href="#loading-data-to-non-lazy-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Non-Lazy MV Datamap</h3>
-<p>In case of WITHOUT DEFERRED REBUILD, for existing table with loaded data, data load to MV table will
-be triggered by the CREATE DATAMAP statement when user creates the MV table.
-For incremental loads to main table, data to datamap will be loaded once the corresponding main
-table load is completed.</p>
-<h3>
-<a id="loading-data-to-lazy-mv-datamap" class="anchor" href="#loading-data-to-lazy-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Lazy MV Datamap</h3>
-<p>In case of WITH DEFERRED REBUILD, data load to MV table will be triggered by the <a href="./datamap-management.html#manual-refresh">Manual Refresh</a>
-command. MV datamap will be in DISABLED state in below scenarios,</p>
-<ul>
-<li>when mv datamap is created</li>
-<li>when data of main table and datamap are not in sync</li>
-</ul>
-<p>User should fire REBUILD DATAMAP command to sync all segments of main table with datamap table and
-which ENABLES the datamap for query</p>
-<h3>
-<a id="loading-data-to-multiple-mvs" class="anchor" href="#loading-data-to-multiple-mvs" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Multiple MV's</h3>
-<p>During load to main table, if anyone of the load to datamap table fails, then that corresponding
-datamap will be DISABLED and load to other datamaps mapped to main table will continue. User can
-fire REBUILD DATAMAP command to sync or else the subsequent table load will load the old failed
-loads along with current load and enable the disabled datamap.</p>
+<a id="loading-data-to-secondary-index-tables" class="anchor" href="#loading-data-to-secondary-index-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading data to Secondary Index table(s).</h3>
+<p><em>case1:</em> When the SI table is created and the main table does not have any data. In this case every
+consecutive load to the main table, will load data to the SI table once the main table data load is finished.</p>
+<p><em>case2:</em> When the SI table is created and the main table already contains some data, then SI creation will
+also load data to the SI table with the same number of segments as the main table. Thereafter, consecutive load to
+the main table will also load data to the SI table.</p>
<p><strong>NOTE</strong>:</p>
<ul>
-<li>In case of InsertOverwrite/Update operation on parent table, all segments of datamap table will
-be MARKED_FOR_DELETE and reload to datamap table will happen by REBUILD DATAMAP, in case of Lazy
-mv datamap/ once InsertOverwrite/Update operation on parent table is finished, in case of
-Non-Lazy mv.</li>
-<li>In case of full scan query, Data Size and Index Size of main table and child table will not the
-same, as main table and child table has different column names.</li>
+<li>In case of data load failure to the SI table, then we make the SI table disable by setting a hive serde
+property. The subsequent main table load will load the old failed loads along with current load and
+makes the SI table enable and available for query.</li>
</ul>
<h2>
<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying data</h2>
-<p>As a technique for query acceleration, MV tables cannot be queried directly.
-Queries are to be made on main table. While doing query planning, internally CarbonData will check
-associated mv datamap tables with the main table, and do query plan transformation accordingly.</p>
-<p>User can verify whether a query can leverage mv datamap table or not by executing <code>EXPLAIN</code>
-command, which will show the transformed logical plan, and thus user can check whether mv datamap
-table is selected.</p>
+<p>Direct query can be made on SI tables to check the data present in position reference columns.
+When a filter query is fired, and if the filter column is a secondary index column, then plan is
+transformed accordingly to hit the SI table first to make better pruning with the main table and in turn
+helps for faster query results.</p>
+<p>Users can verify whether a query can leverage the SI table or not by executing the <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus users can check whether the SI table
+is selected.</p>
<h2>
-<a id="compacting-mv-datamap" class="anchor" href="#compacting-mv-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap</h2>
+<a id="compacting-si-table" class="anchor" href="#compacting-si-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting SI table</h2>
<h3>
-<a id="compacting-mv-datamap-table-through-main-table-compaction" class="anchor" href="#compacting-mv-datamap-table-through-main-table-compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap table through Main Table compaction</h3>
+<a id="compacting-si-table-table-through-main-table-compaction" class="anchor" href="#compacting-si-table-table-through-main-table-compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting SI table table through Main Table compaction</h3>
<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>)[COMPACTION TYPE-> MINOR/MAJOR] on main table will
-automatically compact the mv datamap tables created on the main table, once compaction on main table
-is done.</p>
+automatically delete all the old segments of SI and creates a new segment with same name as main
+table compacted segment and loads data to it.</p>
<h3>
-<a id="compacting-mv-datamap-table-through-ddl-command" class="anchor" href="#compacting-mv-datamap-table-through-ddl-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting MV datamap table through DDL command</h3>
-<p>Compaction on mv datamap can be triggered by running the following DDL command(supported only for mv).</p>
-<pre><code>ALTER DATAMAP datamap_name COMPACT 'COMPACTION_TYPE'
+<a id="compacting-si-tables-individual-segments-through-refresh-index-command" class="anchor" href="#compacting-si-tables-individual-segments-through-refresh-index-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting SI table's individual segment(s) through REFRESH INDEX command</h3>
+<p>Where there are so many small files present in the SI table, then we can use the REFRESH INDEX command to
+compact the files within an SI segment to avoid many small files.</p>
+<pre><code>REFRESH INDEX sales_index ON TABLE sales
+</code></pre>
+<p>This command merges data files in each segment of the SI table.</p>
+<pre><code>REFRESH INDEX sales_index ON TABLE sales WHERE SEGMENT.ID IN(1)
</code></pre>
+<p>This command merges data files within a specified segment of the SI table.</p>
<h2>
-<a id="data-management-with-mv-tables" class="anchor" href="#data-management-with-mv-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with mv tables</h2>
-<p>In current implementation, data consistency needs to be maintained for both main table and mv datamap
-tables. Once there is mv datamap table created on the main table, following command on the main
-table is not supported:</p>
-<ol>
-<li>Data management command: <code>DELETE SEGMENT</code>.</li>
-<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>, <code>ALTER TABLE CHANGE DATATYPE</code>,
-<code>ALTER TABLE RENAME</code>, <code>ALTER COLUMN RENAME</code>. Note that adding a new column is supported, and for
-dropping columns and change datatype command, CarbonData will check whether it will impact the
-mv datamap table, if not, the operation is allowed, otherwise operation will be rejected by
-throwing exception.</li>
-<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>. Note that dropping a partition
-will be allowed only if partition is participating in all datamaps associated with main table.
-Drop Partition is not allowed, if any mv datamap is associated with more than one parent table.
-Drop Partition directly on datamap table is not allowed.</li>
-<li>Complex Datatype's for mv datamap is not supported.</li>
-</ol>
-<p>However, there is still way to support these operations on main table, in current CarbonData
-release, user can do as following:</p>
-<ol>
-<li>Remove the mv datamap table by <code>DROP DATAMAP</code> command</li>
-<li>Carry out the data management operation on main table</li>
-<li>Create the mv datamap table again by <code>CREATE DATAMAP</code> command
-Basically, user can manually trigger the operation by re-building the datamap.</li>
-</ol>
+<a id="how-to-skip-secondary-index" class="anchor" href="#how-to-skip-secondary-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How to skip Secondary Index?</h2>
+<p>When Secondary indexes are created on a table(s), data fetching happens from secondary
+indexes created on the main tables for better performance. But sometimes, data fetching from the
+secondary index might degrade query performance in cases where the data is sparse and most of the
+blocklets need to be scanned. So to avoid such secondary indexes, we use NI as a function on filters
+within WHERE clause.</p>
+<pre><code>SELECT country, sex from sales where NI(user_id = 'xxx')
+</code></pre>
+<p>The above query ignores column <code>user_id</code> from the secondary index and fetches data from the main table.</p>
+<h2>
+<a id="ddls-on-secondary-index" class="anchor" href="#ddls-on-secondary-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DDLs on Secondary Index</h2>
+<h3>
+<a id="show-index-command" class="anchor" href="#show-index-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Show index Command</h3>
+<p>This command is used to get information about all the secondary indexes on a table.</p>
+<p>Syntax</p>
+<pre><code>SHOW INDEXES ON [TABLE] [db_name.]table_name
+</code></pre>
+<h3>
+<a id="drop-index-command" class="anchor" href="#drop-index-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Drop index Command</h3>
+<p>This command is used to drop an existing secondary index on a table</p>
+<p>Syntax</p>
+<pre><code>DROP INDEX [IF EXISTS] index_name ON [TABLE] [db_name.]table_name
+</code></pre>
+<h3>
+<a id="register-index-command" class="anchor" href="#register-index-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Register index Command</h3>
+<p>This command registers the secondary index with the main table in case of compatibility scenarios
+where we have old stores.</p>
+<p>Syntax</p>
+<pre><code>REGISTER INDEX TABLE index_name ON [TABLE] [db_name.]table_name
+</code></pre>
<script>
$(function() {
// Show selected style on nav item
$('.b-nav__datamap').addClass('selected');
-
+
if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
// Display datamap subnav items
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
diff --git a/content/security.html b/content/security.html
index 67e48db..b018008 100644
--- a/content/security.html
+++ b/content/security.html
@@ -45,6 +45,9 @@
aria-expanded="false">Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -66,9 +69,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
diff --git a/content/segment-management-on-carbondata.html b/content/segment-management-on-carbondata.html
index 3fa600b..fc93236 100644
--- a/content/segment-management-on-carbondata.html
+++ b/content/segment-management-on-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -243,15 +252,91 @@ concept which helps to maintain consistency of data and easy transaction managem
<h3>
<a id="show-segment" class="anchor" href="#show-segment" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SHOW SEGMENT</h3>
<p>This command is used to list the segments of CarbonData table.</p>
-<pre><code>SHOW [HISTORY] SEGMENTS FOR TABLE [db_name.]table_name LIMIT number_of_segments
+<pre><code>SHOW [HISTORY] SEGMENTS
+[FOR TABLE | ON] [db_name.]table_name
+[AS (select query from table_name_segments)]
</code></pre>
+<p>By default, SHOW SEGMENT command will return following fields:</p>
+<ul>
+<li>
+<p>Segment ID</p>
+</li>
+<li>
+<p>Segment Status</p>
+</li>
+<li>
+<p>Load Start Time</p>
+</li>
+<li>
+<p>Load Time Taken</p>
+</li>
+<li>
+<p>Partition</p>
+</li>
+<li>
+<p>Data Size</p>
+</li>
+<li>
+<p>Index Size</p>
<p>Example:
Show visible segments</p>
-<pre><code>SHOW SEGMENTS FOR TABLE CarbonDatabase.CarbonTable LIMIT 4
+<pre><code>SHOW SEGMENTS ON CarbonDatabase.CarbonTable
</code></pre>
<p>Show all segments, include invisible segments</p>
-<pre><code>SHOW HISTORY SEGMENTS FOR TABLE CarbonDatabase.CarbonTable LIMIT 4
+<pre><code>SHOW HISTORY SEGMENTS ON CarbonDatabase.CarbonTable
+</code></pre>
+<p>When more detail of the segment is required, user can issue SHOW SEGMENT by query.</p>
+<p>The query should be against table name with '_segments' appended and select from following fields:</p>
+</li>
+<li>
+<p>id: String, the id of the segment</p>
+</li>
+<li>
+<p>status: String, status of the segment</p>
+</li>
+<li>
+<p>loadStartTime: String, loading start time</p>
+</li>
+<li>
+<p>loadEndTime: String, loading end time</p>
+</li>
+<li>
+<p>timeTakenMs: Long, time spent in loading of the segment in milliseconds</p>
+</li>
+<li>
+<p>partitions: String array, partition key and values</p>
+</li>
+<li>
+<p>dataSize: Long, data size in bytes</p>
+</li>
+<li>
+<p>indexSize: Long, index size in bytes</p>
+</li>
+<li>
+<p>mergedToId: String, the target segment that this segment has been compacted</p>
+</li>
+<li>
+<p>format: String, data format of the segment</p>
+</li>
+<li>
+<p>path: String, in case of external segment this will be the path of the segment, otherwise it is null</p>
+</li>
+<li>
+<p>segmentFileName: String, name of the segment file</p>
+<p>Example:</p>
+<pre><code>SHOW SEGMENTS ON CarbonTable AS
+SELECT * FROM CarbonTable_segments
+
+SHOW SEGMENTS ON CarbonTable AS
+SELECT id, dataSize FROM CarbonTable_segments
+WHERE status='Success'
+ORDER BY dataSize
+
+SHOW SEGMENTS ON CarbonTable AS
+SELECT avg(timeTakenMs) FROM CarbonTable_segments
</code></pre>
+</li>
+</ul>
<h3>
<a id="delete-segment-by-id" class="anchor" href="#delete-segment-by-id" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DELETE SEGMENT BY ID</h3>
<p>This command is used to delete segment by using the segment ID. Each segment has a unique segment ID associated with it.
@@ -286,30 +371,30 @@ The segment created before the particular date will be removed from the specific
</code></pre>
<p><strong>NOTE:</strong>
carbon.input.segments: Specifies the segment IDs to be queried. This property allows you to query specified segments of the specified table. The CarbonScan will read data from specified segments only.</p>
-<p>If user wants to query with segments reading in multi threading mode, then CarbonSession. threadSet can be used instead of SET query.</p>
+<p>If user wants to query with segments reading in multi-threading mode, then CarbonSession.threadSet can be used instead of SET query.</p>
<pre><code>CarbonSession.threadSet ("carbon.input.segments.<database_name>.<table_name>","<list of segment IDs>");
</code></pre>
<p>Reset the segment IDs</p>
<pre><code>SET carbon.input.segments.<database_name>.<table_name> = *;
</code></pre>
-<p>If user wants to query with segments reading in multi threading mode, then CarbonSession. threadSet can be used instead of SET query.</p>
+<p>If user wants to query with segments reading in multi-threading mode, then CarbonSession.threadSet can be used instead of SET query.</p>
<pre><code>CarbonSession.threadSet ("carbon.input.segments.<database_name>.<table_name>","*");
</code></pre>
<p><strong>Examples:</strong></p>
<ul>
-<li>Example to show the list of segment IDs,segment status, and other required details and then specify the list of segments to be read.</li>
+<li>Example to show the list of segment IDs, segment status, and other required details and then specify the list of segments to be read.</li>
</ul>
<pre><code>SHOW SEGMENTS FOR carbontable1;
SET carbon.input.segments.db.carbontable1 = 1,3,9;
</code></pre>
<ul>
-<li>Example to query with segments reading in multi threading mode:</li>
+<li>Example to query with segments reading in multi-threading mode:</li>
</ul>
<pre><code>CarbonSession.threadSet ("carbon.input.segments.db.carbontable_Multi_Thread","1,3");
</code></pre>
<ul>
-<li>Example for threadset in multithread environment (following shows how it is used in Scala code):</li>
+<li>Example for threadset in multi-thread environment (following shows how it is used in Scala code):</li>
</ul>
<pre><code>def main(args: Array[String]) {
Future {
diff --git a/src/main/webapp/bloomfilter-datamap-guide.html b/content/spatial-index-guide.html
similarity index 61%
copy from src/main/webapp/bloomfilter-datamap-guide.html
copy to content/spatial-index-guide.html
index 2231c50..88e5fee 100644
--- a/src/main/webapp/bloomfilter-datamap-guide.html
+++ b/content/spatial-index-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,161 +240,108 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-bloomfilter-datamap" class="anchor" href="#carbondata-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter DataMap</h1>
+<a id="what-is-spatial-index" class="anchor" href="#what-is-spatial-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>What is spatial index</h1>
+<p><a href="https://gistbok.ucgis.org/topic-keywords/indexing" target=_blank rel="nofollow">A spatial index</a> is a data structure that allows for accessing a spatial object efficiently. It is a common technique used by spatial databases. Without indexing, any search for a feature would require a "sequential scan" of every record in the database, resulting in much longer processing time. In a spatial index construction process, the minimum bounding rectangle serves as an object approxi [...]
+<h1>
+<a id="how-does-carbondata-implement-spatial-index" class="anchor" href="#how-does-carbondata-implement-spatial-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How does CarbonData implement spatial index</h1>
+<p>There are many open source implementations for spatial indexing and to process spatial queries. CarbonData implements a different way of spatial index. Its core idea is to use the raster data. Raster is made up of matrix of cells organized into rows and columns(called a grid). Each cell represents a coordinate. The index for the coordinate is generated using longitude and latitude, like the <a href="https://en.wikipedia.org/wiki/Z-order_curve" rel="nofollow">Z order curve</a>.</p>
+<p>CarbonData rasterize the user data during data load into segments. A set of latitude and longitude represents a grid range. The size of the grid can be configured. Hence, the coordinates loaded are often discrete and not continuous.</p>
+<p>Below figure shows the relationship between the grid and the points residing in it. Black point represents the center point of the grid, and the red points are the coordinates at the arbitrary positions inside the grid. The red points can be replaced by the center point of the grid to indicate that the points lies within the grid. During data load, CarbonData generates an index for the coordinate according to row and column of the grid(in the raster) where that coordinate lies. These [...]
+<p><a href="../docs/images/spatial-index-1.png?raw=true" target="_blank" rel="noopener noreferrer"><img src="https://github.com/apache/carbondata/blob/master/docs/images/spatial-index-1.png?raw=true" alt="File Directory Structure" style="max-width:100%;"></a></p>
+<p>Carbon supports Polygon User Defined Function(UDF) as filter condition in the query to return all the data points lying within it. Polygon UDF takes multiple points(i.e., pair of longitude and latitude) separated by a comma. Longitude and latitude in the pair are separated by a space. The first and last points in the polygon must be same to form a closed loop. CarbonData builds a quad tree using this polygon and spatial region information passed while creating a table. The nodes in th [...]
+The main reasons for faster query response are as follows :</p>
<ul>
-<li><a href="#datamap-management">DataMap Management</a></li>
-<li><a href="#bloomfilter-datamap-introduction">BloomFilter Datamap Introduction</a></li>
-<li><a href="#loading-data">Loading Data</a></li>
-<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#data-management-with-bloomfilter-datamap">Data Management</a></li>
-<li><a href="#useful-tips">Useful Tips</a></li>
+<li>Data is sorted based on the index values.</li>
+<li>Polygon UDF filter is pushed down from engine to the carbon layer such that CarbonData scans only matched blocklets avoiding full scan.</li>
</ul>
-<h4>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
-<p>Creating BloomFilter DataMap</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-ON TABLE main_table
-USING 'bloomfilter'
-DMPROPERTIES ('index_columns'='city, name', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
-</code></pre>
-<p>Dropping Specified DataMap</p>
-<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
-ON TABLE main_table
-</code></pre>
-<p>Showing all DataMaps on this table</p>
-<pre><code>SHOW DATAMAP
-ON TABLE main_table
-</code></pre>
-<p>Disable DataMap</p>
-<blockquote>
-<p>The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
-</blockquote>
-<pre><code>// disable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = false
-// enable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = true
-</code></pre>
+<p><a href="../docs/images/spatial-index-2.png?raw=true" target="_blank" rel="noopener noreferrer"><img src="https://github.com/apache/carbondata/blob/master/docs/images/spatial-index-2.png?raw=true" alt="File Directory Structure" style="max-width:100%;"></a></p>
+<h1>
+<a id="installation-and-deployment" class="anchor" href="#installation-and-deployment" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation and Deployment</h1>
+<p>Geo is a separate module in the Project. It can be included or excluded from the project build based on the requirement.</p>
<h2>
-<a id="bloomfilter-datamap-introduction" class="anchor" href="#bloomfilter-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter DataMap Introduction</h2>
-<p>A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.
-Carbondata introduced BloomFilter as an index datamap to enhance the performance of querying with precise value.
-It is well suitable for queries that do precise match on high cardinality columns(such as Name/ID).
-Internally, CarbonData maintains a BloomFilter per blocklet for each index column to indicate that whether a value of the column is in this blocklet.
-Just like the other datamaps, BloomFilter datamap is managed along with main tables by CarbonData.
-User can create BloomFilter datamap on specified columns with specified BloomFilter configurations such as size and probability.</p>
-<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
-<pre><code>CREATE TABLE datamap_test (
- id string,
- name string,
- age int,
- city string,
- country string)
-STORED AS carbondata
-TBLPROPERTIES('SORT_COLUMNS'='id')
-</code></pre>
-<p>In the above example, <code>id</code> and <code>name</code> are high cardinality columns
-and we always query on <code>id</code> and <code>name</code> with precise value.
-since <code>id</code> is in the sort_columns and it is orderd,
-query on it will be fast because CarbonData can skip all the irrelative blocklets.
-But queries on <code>name</code> may be bad since the blocklet minmax may not help,
-because in each blocklet the range of the value of <code>name</code> may be the same -- all from A* to z*.
-In this case, user can create a BloomFilter DataMap on column <code>name</code>.
-Moreover, user can also create a BloomFilter DataMap on the sort_columns.
-This is useful if user has too many segments and the range of the value of sort_columns are almost the same.</p>
-<p>User can create BloomFilter DataMap using the Create DataMap DDL:</p>
-<pre><code>CREATE DATAMAP dm
-ON TABLE datamap_test
-USING 'bloomfilter'
-DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001', 'BLOOM_COMPRESS'='true')
+<a id="basic-command" class="anchor" href="#basic-command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Basic Command</h2>
+<h3>
+<a id="create-table" class="anchor" href="#create-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Table</h3>
+<p>Create table with spatial index table properties</p>
+<pre><code>create table source_index(id BIGINT, latitude long, longitude long) stored by 'carbondata' TBLPROPERTIES (
+'SPATIAL_INDEX'='mygeohash',
+'SPATIAL_INDEX.mygeohash.type'='geohash',
+'SPATIAL_INDEX.mygeohash.sourcecolumns'='longitude, latitude',
+'SPATIAL_INDEX.mygeohash.originLatitude'='19.832277',
+'SPATIAL_INDEX.mygeohash.gridSize'='50',
+'SPATIAL_INDEX.mygeohash.minLongitude'='1.811865',
+'SPATIAL_INDEX.mygeohash.maxLongitude'='2.782233',
+'SPATIAL_INDEX.mygeohash.minLatitude'='19.832277',
+'SPATIAL_INDEX.mygeohash.maxLatitude'='20.225281',
+'SPATIAL_INDEX.mygeohash.conversionRatio'='1000000');
</code></pre>
-<p><strong>Properties for BloomFilter DataMap</strong></p>
+<p>Note: <code>mygeohash</code> in the above example represent the index name.</p>
+<h4>
+<a id="list-of-spatial-index-table-properties" class="anchor" href="#list-of-spatial-index-table-properties" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>List of spatial index table properties</h4>
<table>
<thead>
<tr>
-<th>Property</th>
-<th>Is Required</th>
-<th>Default Value</th>
+<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
-<td>INDEX_COLUMNS</td>
-<td>YES</td>
-<td></td>
-<td>Carbondata will generate BloomFilter index on these columns. Queries on these columns are usually like 'COL = VAL'.</td>
+<td>SPATIAL_INDEX</td>
+<td>Used to configure Spatial Index name. This name is appended to <code>SPATIAL_INDEX</code> in the subsequent sub-property configurations. <code>xxx</code> in the below sub-properties refer to index name.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.type</td>
+<td>Type of algorithm for processing spatial data. Currently, supports only 'geohash'.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.sourcecolumns</td>
+<td>longitude and latitude column names as in the table. These columns are used to generate index value for each row.</td>
</tr>
<tr>
-<td>BLOOM_SIZE</td>
-<td>NO</td>
-<td>640000</td>
-<td>This value is internally used by BloomFilter as the number of expected insertions, it will affect the size of BloomFilter index. Since each blocklet has a BloomFilter here, so the default value is the approximate distinct index values in a blocklet assuming that each blocklet contains 20 pages and each page contains 32000 records. The value should be an integer.</td>
+<td>SPATIAL_INDEX.xxx.gridSize</td>
+<td>Grid size of raster data in metres. Currently, spatial index supports raster data.</td>
</tr>
<tr>
-<td>BLOOM_FPP</td>
-<td>NO</td>
-<td>0.00001</td>
-<td>This value is internally used by BloomFilter as the False-Positive Probability, it will affect the size of bloomfilter index as well as the number of hash functions for the BloomFilter. The value should be in the range (0, 1). In one test scenario, a 96GB TPCH customer table with bloom_size=320000 and bloom_fpp=0.00001 will result in 18 false positive samples.</td>
+<td>SPATIAL_INDEX.xxx.minLongitude</td>
+<td>Minimum longitude of the gridded rectangular area.</td>
</tr>
<tr>
-<td>BLOOM_COMPRESS</td>
-<td>NO</td>
-<td>true</td>
-<td>Whether to compress the BloomFilter index files.</td>
+<td>SPATIAL_INDEX.xxx.maxLongitude</td>
+<td>Maximum longitude of the gridded rectangular area.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.minLatitude</td>
+<td>Minimum latitude of the gridded rectangular area.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.maxLatitude</td>
+<td>Maximum latitude of the gridded rectangular area.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.conversionRatio</td>
+<td>Conversion factor. It allows user to translate longitude and latitude to long. For example, if the data to load is longitude = 13.123456, latitude = 101.12356. User can configure conversion ratio sub-property value as 1000000, and change data to load as longitude = 13123456 and latitude = 10112356. Operations on long is much faster compared to floating-point numbers.</td>
+</tr>
+<tr>
+<td>SPATIAL_INDEX.xxx.class</td>
+<td>Optional user custom implementation class. Value is fully qualified class name.</td>
</tr>
</tbody>
</table>
+<h3>
+<a id="select-query" class="anchor" href="#select-query" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Select Query</h3>
+<p>Query with Polygon UDF predicate</p>
+<pre><code>select * from source_index where IN_POLYGON('16.321011 4.123503,16.137676 5.947911,16.560993 5.935276,16.321011 4.123503')
+</code></pre>
<h2>
-<a id="loading-data" class="anchor" href="#loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data</h2>
-<p>When loading data to main table, BloomFilter files will be generated for all the
-index_columns given in DMProperties which contains the blockletId and a BloomFilter for each index column.
-These index files will be written inside a folder named with DataMap name
-inside each segment folders.</p>
-<h2>
-<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying Data</h2>
-<p>User can verify whether a query can leverage BloomFilter DataMap by executing <code>EXPLAIN</code> command,
-which will show the transformed logical plan, and thus user can check whether the BloomFilter DataMap can skip blocklets during the scan.
-If the DataMap does not prune blocklets well, you can try to increase the value of property <code>BLOOM_SIZE</code> and decrease the value of property <code>BLOOM_FPP</code>.</p>
-<h2>
-<a id="data-management-with-bloomfilter-datamap" class="anchor" href="#data-management-with-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management With BloomFilter DataMap</h2>
-<p>Data management with BloomFilter DataMap has no difference with that on Lucene DataMap.
-You can refer to the corresponding section in <code>CarbonData Lucene DataMap</code>.</p>
-<h2>
-<a id="useful-tips" class="anchor" href="#useful-tips" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Useful Tips</h2>
-<ul>
-<li>BloomFilter DataMap is suggested to be created on the high cardinality columns.
-Query conditions on these columns are always simple <code>equal</code> or <code>in</code>,
-such as 'col1=XX', 'col1 in (XX, YY)'.</li>
-<li>We can create multiple BloomFilter DataMaps on one table,
-but we do recommend you to create one BloomFilter DataMap that contains multiple index columns,
-because the data loading and query performance will be better.</li>
-<li>
-<code>BLOOM_FPP</code> is only the expected number from user, the actually FPP may be worse.
-If the BloomFilter DataMap does not work well,
-you can try to increase <code>BLOOM_SIZE</code> and decrease <code>BLOOM_FPP</code> at the same time.
-Notice that bigger <code>BLOOM_SIZE</code> will increase the size of index file
-and smaller <code>BLOOM_FPP</code> will increase runtime calculation while performing query.</li>
-<li>'0' skipped blocklets of BloomFilter DataMap in explain output indicates that
-BloomFilter DataMap does not prune better than Main DataMap.
-(For example since the data is not ordered, a specific value may be contained in many blocklets. In this case, bloom may not work better than Main DataMap.)
-If this occurs very often, it means that current BloomFilter is useless. You can disable or drop it.
-Sometimes we cannot see any pruning result about BloomFilter DataMap in the explain output,
-this indicates that the previous DataMap has pruned all the blocklets and there is no need to continue pruning.</li>
-<li>In some scenarios, the BloomFilter DataMap may not enhance the query performance significantly
-but if it can reduce the number of spark task,
-there is still a chance that BloomFilter DataMap can enhance the performance for concurrent query.</li>
-<li>Note that BloomFilter DataMap will decrease the data loading performance and may cause slightly storage expansion (for DataMap index file).</li>
-</ul>
+<a id="reference" class="anchor" href="#reference" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Reference</h2>
+<pre><code>[1] https://issues.apache.org/jira/browse/CARBONDATA-3548
+[2] https://gistbok.ucgis.org/topic-keywords/indexing
+[3] https://en.wikipedia.org/wiki/Z-order_curve
+</code></pre>
<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
-
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
+// Show selected style on nav item
+$(function() { $('.b-nav__quickstart').addClass('selected'); });
</script></div>
</div>
</div>
diff --git a/content/streaming-guide.html b/content/streaming-guide.html
index c6cf9fc..d72fd63 100644
--- a/content/streaming-guide.html
+++ b/content/streaming-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -266,9 +275,9 @@
</ul>
<h2>
<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2>
-<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
-<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-1.3.0-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
-<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.2</pre></div>
+<p>Download and unzip spark-2.4.5-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
+<p>Package carbon jar, and copy assembly/target/scala-2.11/carbondata_2.11-2.0.0-SNAPSHOT-shade-hadoop2.7.2.jar to $SPARK_HOME/jars</p>
+<div class="highlight highlight-source-shell"><pre>mvn clean package -DskipTests -Pspark-2.4</pre></div>
<p>Start a socket data server in a terminal</p>
<div class="highlight highlight-source-shell"><pre>nc -lk 9099</pre></div>
<p>type some CSV rows as following</p>
@@ -334,7 +343,7 @@
<span class="pl-c"><span class="pl-c">//</span> start new thread to show data</span>
<span class="pl-k">new</span> <span class="pl-en">Thread</span>() {
- <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">run</span>()<span class="pl-k">:</span> <span class="pl-k">Unit</span> <span class="pl-k">=</span> {
+ <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">run</span>()<span class="pl-k">:</span> <span class="pl-en">Unit</span> <span class="pl-k">=</span> {
<span class="pl-k">do</span> {
spark.sql(<span class="pl-s"><span class="pl-pds">"</span>select * from carbon_table<span class="pl-pds">"</span></span>).show(<span class="pl-c1">false</span>)
<span class="pl-en">Thread</span>.sleep(<span class="pl-c1">10000</span>)
@@ -467,8 +476,8 @@ TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span cla
<p>Currently CarbonData support two parsers, as following:</p>
<p><strong>1. org.apache.carbondata.streaming.parser.CSVStreamParserImp</strong>: This parser gets a line data(String type) from the first index of InternalRow and converts this String to Object[].</p>
<p><strong>2. org.apache.carbondata.streaming.parser.RowStreamParserImp</strong>: This is the default stream parser, it will auto convert InternalRow to Object[] according to schema of this <code>DataSet</code>, for example:</p>
-<div class="highlight highlight-source-scala"><pre> <span class="pl-k">case</span> <span class="pl-k">class</span> <span class="pl-en">FileElement</span>(<span class="pl-v">school</span>: <span class="pl-en">Array</span>[<span class="pl-k">String</span>], <span class="pl-v">age</span>: <span class="pl-k">Int</span>)
- <span class="pl-k">case</span> <span class="pl-k">class</span> <span class="pl-en">StreamData</span>(<span class="pl-v">id</span>: <span class="pl-k">Int</span>, <span class="pl-v">name</span>: <span class="pl-k">String</span>, <span class="pl-v">city</span>: <span class="pl-k">String</span>, <span class="pl-v">salary</span>: <span class="pl-k">Float</span>, <span class="pl-v">file</span>: <span class="pl-en">FileElement</span>)
+<div class="highlight highlight-source-scala"><pre> <span class="pl-k">case</span> <span class="pl-k">class</span> <span class="pl-en">FileElement</span>(<span class="pl-v">school</span>: <span class="pl-en">Array</span>[<span class="pl-en">String</span>], <span class="pl-v">age</span>: <span class="pl-en">Int</span>)
+ <span class="pl-k">case</span> <span class="pl-k">class</span> <span class="pl-en">StreamData</span>(<span class="pl-v">id</span>: <span class="pl-en">Int</span>, <span class="pl-v">name</span>: <span class="pl-en">String</span>, <span class="pl-v">city</span>: <span class="pl-en">String</span>, <span class="pl-v">salary</span>: <span class="pl-en">Float</span>, <span class="pl-v">file</span>: <span class="pl-en">FileElement</span>)
...
<span class="pl-k">var</span> <span class="pl-smi">qry</span><span class="pl-k">:</span> <span class="pl-en">StreamingQuery</span> <span class="pl-k">=</span> <span class="pl-c1">null</span>
@@ -477,7 +486,7 @@ TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span cla
.option(<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>localhost<span class="pl-pds">"</span></span>)
.option(<span class="pl-s"><span class="pl-pds">"</span>port<span class="pl-pds">"</span></span>, <span class="pl-c1">9099</span>)
.load()
- .as[<span class="pl-k">String</span>]
+ .as[<span class="pl-en">String</span>]
.map(_.split(<span class="pl-s"><span class="pl-pds">"</span>,<span class="pl-pds">"</span></span>))
.map { fields <span class="pl-k">=></span> {
<span class="pl-k">val</span> <span class="pl-smi">tmp</span> <span class="pl-k">=</span> fields(<span class="pl-c1">4</span>).split(<span class="pl-s"><span class="pl-pds">"</span><span class="pl-cce">\\</span>$<span class="pl-pds">"</span></span>)
@@ -506,7 +515,7 @@ TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span cla
<span class="pl-k">class</span> <span class="pl-en">XXXStreamParserImp</span> <span class="pl-k">extends</span> <span class="pl-e">CarbonStreamParser</span> {
- <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">initialize</span>(<span class="pl-v">configuration</span>: <span class="pl-en">Configuration</span>, <span class="pl-v">structType</span>: <span class="pl-en">StructType</span>)<span class="pl-k">:</span> <span class="pl-k">Unit</span> <span class="pl-k">=</span> {
+ <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">initialize</span>(<span class="pl-v">configuration</span>: <span class="pl-en">Configuration</span>, <span class="pl-v">structType</span>: <span class="pl-en">StructType</span>)<span class="pl-k">:</span> <span class="pl-en">Unit</span> <span class="pl-k">=</span> {
<span class="pl-c"><span class="pl-c">//</span> user can get the properties from "configuration"</span>
}
@@ -514,7 +523,7 @@ TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span cla
<span class="pl-c"><span class="pl-c">//</span> convert InternalRow to Object[](Array[Object] in Scala) </span>
}
- <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">close</span>()<span class="pl-k">:</span> <span class="pl-k">Unit</span> <span class="pl-k">=</span> {
+ <span class="pl-k">override</span> <span class="pl-k">def</span> <span class="pl-en">close</span>()<span class="pl-k">:</span> <span class="pl-en">Unit</span> <span class="pl-k">=</span> {
}
}
</pre></div>
@@ -529,8 +538,8 @@ TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span cla
<ol>
<li>reject set streaming property from true to false.</li>
<li>reject UPDATE/DELETE command on the streaming table.</li>
-<li>reject create pre-aggregation DataMap on the streaming table.</li>
-<li>reject add the streaming property on the table with pre-aggregation DataMap.</li>
+<li>reject create MV on the streaming table.</li>
+<li>reject add the streaming property on the table with MV.</li>
<li>if the table has dictionary columns, it will not support concurrent data loading.</li>
<li>block delete "streaming" segment while the streaming ingestion is running.</li>
<li>block drop the streaming table while the streaming ingestion is running.</li>
diff --git a/content/supported-data-types-in-carbondata.html b/content/supported-data-types-in-carbondata.html
index c792623..e1af375 100644
--- a/content/supported-data-types-in-carbondata.html
+++ b/content/supported-data-types-in-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html
deleted file mode 100644
index 1295b03..0000000
--- a/content/timeseries-datamap-guide.html
+++ /dev/null
@@ -1,380 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
- <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
- <title>CarbonData</title>
- <style>
-
- </style>
- <!-- Bootstrap -->
-
- <link rel="stylesheet" href="css/bootstrap.min.css">
- <link href="css/style.css" rel="stylesheet">
- <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
- <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
- <!--[if lt IE 9]>
- <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
- <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js"></script>
- <![endif]-->
- <script src="js/jquery.min.js"></script>
- <script src="js/bootstrap.min.js"></script>
- <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js"></script>
-
-
-</head>
-<body>
-<header>
- <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
- <div class="container">
- <div class="navbar-header">
- <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse"
- class="navbar-toggle collapsed" type="button">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a href="index.html" class="logo">
- <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/>
- </a>
- </div>
- <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
- <ul class="nav navbar-nav navbar-right navlist-custom">
- <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a>
- </li>
- <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false"> Download <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
- target="_blank">Apache CarbonData 1.6.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.0/"
- target="_blank">Apache CarbonData 1.6.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.4/"
- target="_blank">Apache CarbonData 1.5.4</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.3/"
- target="_blank">Apache CarbonData 1.5.3</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.2/"
- target="_blank">Apache CarbonData 1.5.2</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"
- target="_blank">Apache CarbonData 1.5.1</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
- target="_blank">Apache CarbonData 1.5.0</a></li>
- <li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
- target="_blank">Release Archive</a></li>
- </ul>
- </li>
- <li><a href="documentation.html" class="active">Documentation</a></li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
- aria-expanded="false">Community <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"
- target="_blank">Contributing to CarbonData</a></li>
- <li>
- <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"
- target="_blank">Release Guide</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"
- target="_blank">Project PMC and Committers</a></li>
- <li>
- <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"
- target="_blank">CarbonData Meetups</a></li>
- <li><a href="security.html">Apache CarbonData Security</a></li>
- <li><a href="https://issues.apache.org/jira/browse/CARBONDATA" target="_blank">Apache
- Jira</a></li>
- <li><a href="videogallery.html">CarbonData Videos </a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="http://www.apache.org/" class="apache_link hidden-xs dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li class="dropdown">
- <a href="http://www.apache.org/" class="hidden-lg hidden-md hidden-sm dropdown-toggle"
- data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/" target="_blank">Apache Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html"
- target="_blank">Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
- </ul>
- </li>
-
- <li>
- <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a>
-
- </li>
-
- </ul>
- </div><!--/.nav-collapse -->
- <div id="search-box">
- <form method="get" action="http://www.google.com/search" target="_blank">
- <div class="search-block">
- <table border="0" cellpadding="0" width="100%">
- <tr>
- <td style="width:80%">
- <input type="text" name="q" size=" 5" maxlength="255" value=""
- class="search-input" placeholder="Search...." required/>
- </td>
- <td style="width:20%">
- <input type="submit" value="Search"/></td>
- </tr>
- <tr>
- <td align="left" style="font-size:75%" colspan="2">
- <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/>
- <span style=" position: relative; top: -3px;"> Only search for CarbonData</span>
- </td>
- </tr>
- </table>
- </div>
- </form>
- </div>
- </div>
- </nav>
-</header> <!-- end Header part -->
-
-<div class="fixed-padding"></div> <!-- top padding with fixde header -->
-
-<section><!-- Dashboard nav -->
- <div class="container-fluid q">
- <div class="col-sm-12 col-md-12 maindashboard">
- <div class="verticalnavbar">
- <nav class="b-sticky-nav">
- <div class="nav-scroller">
- <div class="nav__inner">
- <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a>
- <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a>
- <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a>
- <a class="nav__item nav__sub__item" href="./ddl-of-carbondata.html">DDL</a>
- <a class="nav__item nav__sub__item" href="./dml-of-carbondata.html">DML</a>
- <a class="nav__item nav__sub__item" href="./streaming-guide.html">Streaming</a>
- <a class="nav__item nav__sub__item" href="./configuration-parameters.html">Configuration</a>
- <a class="nav__item nav__sub__item" href="./datamap-developer-guide.html">Datamaps</a>
- <a class="nav__item nav__sub__item" href="./supported-data-types-in-carbondata.html">Data Types</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
- </div>
-
- <div class="nav__item nav__item__with__subs">
- <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a>
- <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a>
- <a class="nav__item nav__sub__item" href="./csdk-guide.html">C++ SDK</a>
- </div>
-
- <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
- <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
- <a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
- <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
- <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
- <a class="b-nav__security nav__item" href="./security.html">Security</a>
- <a class="b-nav__release nav__item" href="./release-guide.html">Release Guide</a>
- </div>
- </div>
- <div class="navindicator">
- <div class="b-nav__intro navindicator__item"></div>
- <div class="b-nav__quickstart navindicator__item"></div>
- <div class="b-nav__uses navindicator__item"></div>
- <div class="b-nav__docs navindicator__item"></div>
- <div class="b-nav__datamap navindicator__item"></div>
- <div class="b-nav__api navindicator__item"></div>
- <div class="b-nav__perf navindicator__item"></div>
- <div class="b-nav__s3 navindicator__item"></div>
- <div class="b-nav__indexserver navindicator__item"></div>
- <div class="b-nav__faq navindicator__item"></div>
- <div class="b-nav__contri navindicator__item"></div>
- <div class="b-nav__security navindicator__item"></div>
- </div>
- </nav>
- </div>
- <div class="mdcontent">
- <section>
- <div style="padding:10px 15px;">
- <div id="viewpage" name="viewpage">
- <div class="row">
- <div class="col-sm-12 col-md-12">
- <div>
-<h1>
-<a id="carbondata-timeseries-datamap" class="anchor" href="#carbondata-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Timeseries DataMap</h1>
-<ul>
-<li><a href="#timeseries-datamap-introduction-alpha-feature">Timeseries DataMap Introduction</a></li>
-<li><a href="#compacting-timeseries-datamp">Compaction</a></li>
-<li><a href="#data-management-on-timeseries-datamap">Data Management</a></li>
-</ul>
-<h2>
-<a id="timeseries-datamap-introduction-alpha-feature" class="anchor" href="#timeseries-datamap-introduction-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap Introduction (Alpha Feature)</h2>
-<p>Timeseries DataMap is a pre-aggregate table implementation based on 'pre-aggregate' DataMap.
-Difference is that Timeseries DataMap has built-in understanding of time hierarchy and
-levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension
-for query.</p>
-<p><strong>CAUTION:</strong> Current version of CarbonData does not support roll-up.It will be implemented in future versions.</p>
-<p>The data loading, querying, compaction command and its behavior is the same as preaggregate DataMap.
-Please refer to <a href="./preaggregate-datamap-guide.html">Pre-aggregate DataMap</a>
-for more information.</p>
-<p>To use this datamap, user can create multiple timeseries datamap on the main table which has
-a <em>event_time</em> column, one datamap for one time granularity.</p>
-<p>For example, below statement effectively create multiple pre-aggregate tables on main table called
-<strong>timeseries</strong></p>
-<pre><code>CREATE DATAMAP agg_year
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'year_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-
-CREATE DATAMAP agg_month
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'month_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-
-CREATE DATAMAP agg_day
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'day_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-
-CREATE DATAMAP agg_sales_hour
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'hour_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-
-CREATE DATAMAP agg_minute
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'minute_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-</code></pre>
-<p>For querying timeseries data, Carbondata has builtin support for following time related UDF</p>
-<pre><code>timeseries(timeseries column name, 'aggregation level')
-</code></pre>
-<pre><code>SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales GROUP BY timeseries(order_time,'hour')
-</code></pre>
-<p>It is <strong>not necessary</strong> to create pre-aggregate tables for each granularity unless required for
-query.</p>
-<p>For Example: For main table <strong>sales</strong> , if following timeseries datamaps were created for day
-level and hour level pre-aggregate</p>
-<pre><code>CREATE DATAMAP agg_day
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'day_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-
-CREATE DATAMAP agg_sales_hour
-ON TABLE sales
-USING "timeseries"
-DMPROPERTIES (
- 'event_time'='order_time',
- 'hour_granularity'='1',
-) AS
-SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
- avg(price) FROM sales GROUP BY order_time, country, sex
-</code></pre>
-<p>Queries like below will not be rolled-up and hit the main table</p>
-<pre><code>Select timeseries(order_time, 'month'), sum(quantity) from sales group by timeseries(order_time,
- 'month')
-
-Select timeseries(order_time, 'year'), sum(quantity) from sales group by timeseries(order_time,
- 'year')
-</code></pre>
-<p>NOTE (<strong>RESTRICTION</strong>):</p>
-<ul>
-<li>Only value of 1 is supported for hierarchy levels. Other hierarchy levels will be supported in
-the future CarbonData release.</li>
-<li>timeseries datamap for the desired levels needs to be created one after the other</li>
-<li>timeseries datamaps created for each level needs to be dropped separately</li>
-</ul>
-<h2>
-<a id="compacting-timeseries-datamp" class="anchor" href="#compacting-timeseries-datamp" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compacting timeseries datamp</h2>
-<p>Refer to Compaction section in <a href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
-Same applies to timeseries datamap.</p>
-<h2>
-<a id="data-management-on-timeseries-datamap" class="anchor" href="#data-management-on-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management on timeseries datamap</h2>
-<p>Refer to Data Management section in <a href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
-Same applies to timeseries datamap.</p>
-<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
-
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
-</script></div>
-</div>
-</div>
-</div>
-<div class="doc-footer">
- <a href="#top" class="scroll-top">Top</a>
-</div>
-</div>
-</section>
-</div>
-</div>
-</div>
-</section><!-- End systemblock part -->
-<script src="js/custom.js"></script>
-</body>
-</html>
\ No newline at end of file
diff --git a/content/usecases.html b/content/usecases.html
index 7325d66..2ce3e52 100644
--- a/content/usecases.html
+++ b/content/usecases.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -308,12 +317,6 @@
<tbody>
<tr>
<td>Data Loading</td>
-<td>carbon.graph.rowset.size</td>
-<td>100000</td>
-<td>Based on the size of each row, this determines the memory required during data loading.Higher value leads to increased memory foot print</td>
-</tr>
-<tr>
-<td>Data Loading</td>
<td>carbon.number.of.cores.while.loading</td>
<td>12</td>
<td>More cores can improve data loading speed</td>
@@ -421,12 +424,6 @@
<tbody>
<tr>
<td>Data Loading</td>
-<td>carbon.graph.rowset.size</td>
-<td>100000</td>
-<td>Based on the size of each row, this determines the memory required during data loading.Higher value leads to increased memory foot print</td>
-</tr>
-<tr>
-<td>Data Loading</td>
<td>enable.unsafe.sort</td>
<td>TRUE</td>
<td>Temporary data generated during sort is huge which causes GC bottlenecks. Using unsafe reduces the pressure on GC</td>
@@ -475,7 +472,7 @@
</tr>
<tr>
<td>Data Loading</td>
-<td>sort.inmemory.size.in.mb</td>
+<td>sort.inmemory.size.inmb</td>
<td>92160</td>
<td>Memory allocated to do inmemory sorting. When more memory is available in the node, configuring this will retain more sort blocks in memory so that the merge sort is faster due to no/very less IO</td>
</tr>
@@ -571,10 +568,8 @@
<ul>
<li>Use table block size as 128MB so that pruning is more effective</li>
<li>Use global sort mode so that the data to be fetched are grouped together</li>
-<li>Create pre-aggregate tables for non timestamp based group by queries</li>
-<li>For queries containing group by date, create timeseries based Datamap(pre-aggregate) tables so that the data is rolled up during creation and fetch is faster</li>
+<li>Create Materialized View for aggregation queries</li>
<li>Reduce the Spark shuffle partitions.(In our configuration on 14 node cluster, it was reduced to 35 from default of 200)</li>
-<li>Enable global dictionary for columns which have less cardinalities. Aggregation can be done on encoded data, there by improving the performance</li>
<li>For columns whose cardinality is high,enable the local dictionary so that store size is less and can take dictionary benefit for scan</li>
</ul>
<h2>
diff --git a/content/videogallery.html b/content/videogallery.html
index 23ec4e2..f603c06 100644
--- a/content/videogallery.html
+++ b/content/videogallery.html
@@ -49,6 +49,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -70,9 +73,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf
index eb1c327..970edf4 100644
--- a/src/main/resources/application.conf
+++ b/src/main/resources/application.conf
@@ -6,7 +6,7 @@ fileList=["configuration-parameters",
"supported-data-types-in-carbondata",
"performance-tuning",
"sdk-guide",
- "datamap-developer-guide",
+ "index-developer-guide",
"ddl-of-carbondata",
"dml-of-carbondata",
"documentation",
@@ -22,17 +22,23 @@ fileList=["configuration-parameters",
"alluxio-guide",
"hive-guide",
"index-server",
- "presto-guide"
+
+ "flink-integration-guide",
+ "mv-guide",
+ "prestodb-guide",
+ "prestosql-guide",
+ "scd-and-cdc-guide",
+ "spatial-index-guide",
+
+
]
dataMapFileList=[
- "bloomfilter-datamap-guide",
- "lucene-datamap-guide",
- "preaggregate-datamap-guide",
- "mv-datamap-guide",
- "timeseries-datamap-guide",
- "datamap-management"]
+ "bloomfilter-index-guide",
+ "lucene-index-guide",
+ "secondary-index-guide",
+ "index-management"]
-dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/datamap/"
+dataMapFilesUrl="https://raw.githubusercontent.com/apache/carbondata/master/docs/index/"
fileListToRetain=["quick-start-guide"
]
diff --git a/src/main/scala/html/header.html b/src/main/scala/html/header.html
index 971358b..5425e83 100644
--- a/src/main/scala/html/header.html
+++ b/src/main/scala/html/header.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
diff --git a/src/main/scala/scripts/bloomfilter-datamap-guide b/src/main/scala/scripts/bloomfilter-datamap-guide
deleted file mode 100644
index 8834f00..0000000
--- a/src/main/scala/scripts/bloomfilter-datamap-guide
+++ /dev/null
@@ -1,11 +0,0 @@
-<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
-
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
-</script>
\ No newline at end of file
diff --git a/src/main/scala/scripts/timeseries-datamap-guide b/src/main/scala/scripts/bloomfilter-index-guide
similarity index 100%
rename from src/main/scala/scripts/timeseries-datamap-guide
rename to src/main/scala/scripts/bloomfilter-index-guide
diff --git a/src/main/scala/scripts/datamap-developer-guide b/src/main/scala/scripts/flink-integration-guide
similarity index 100%
copy from src/main/scala/scripts/datamap-developer-guide
copy to src/main/scala/scripts/flink-integration-guide
diff --git a/src/main/scala/scripts/datamap-developer-guide b/src/main/scala/scripts/index-developer-guide
similarity index 100%
copy from src/main/scala/scripts/datamap-developer-guide
copy to src/main/scala/scripts/index-developer-guide
diff --git a/src/main/scala/scripts/preaggregate-datamap-guide b/src/main/scala/scripts/index-management
similarity index 100%
rename from src/main/scala/scripts/preaggregate-datamap-guide
rename to src/main/scala/scripts/index-management
diff --git a/src/main/scala/scripts/lucene-datamap-guide b/src/main/scala/scripts/lucene-index-guide
similarity index 100%
rename from src/main/scala/scripts/lucene-datamap-guide
rename to src/main/scala/scripts/lucene-index-guide
diff --git a/src/main/scala/scripts/mv-datamap-guide b/src/main/scala/scripts/mv-datamap-guide
deleted file mode 100644
index 013400a..0000000
--- a/src/main/scala/scripts/mv-datamap-guide
+++ /dev/null
@@ -1,11 +0,0 @@
-<script>
-$(function() {
- // Show selected style on nav item
- $('.b-nav__datamap').addClass('selected');
-
- if (!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
- // Display datamap subnav items
- $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
- }
-});
-</script>
\ No newline at end of file
diff --git a/src/main/scala/scripts/datamap-developer-guide b/src/main/scala/scripts/mv-guide
similarity index 100%
rename from src/main/scala/scripts/datamap-developer-guide
rename to src/main/scala/scripts/mv-guide
diff --git a/src/main/scala/scripts/presto-guide b/src/main/scala/scripts/prestodb-guide
similarity index 100%
copy from src/main/scala/scripts/presto-guide
copy to src/main/scala/scripts/prestodb-guide
diff --git a/src/main/scala/scripts/presto-guide b/src/main/scala/scripts/prestosql-guide
similarity index 100%
copy from src/main/scala/scripts/presto-guide
copy to src/main/scala/scripts/prestosql-guide
diff --git a/src/main/scala/scripts/presto-guide b/src/main/scala/scripts/scd-and-cdc-guide
similarity index 100%
copy from src/main/scala/scripts/presto-guide
copy to src/main/scala/scripts/scd-and-cdc-guide
diff --git a/src/main/scala/scripts/datamap-management b/src/main/scala/scripts/secondary-index-guide
similarity index 100%
rename from src/main/scala/scripts/datamap-management
rename to src/main/scala/scripts/secondary-index-guide
diff --git a/src/main/scala/scripts/presto-guide b/src/main/scala/scripts/spatial-index-guide
similarity index 100%
rename from src/main/scala/scripts/presto-guide
rename to src/main/scala/scripts/spatial-index-guide
diff --git a/src/main/webapp/alluxio-guide.html b/src/main/webapp/alluxio-guide.html
index ead4da4..a470301 100644
--- a/src/main/webapp/alluxio-guide.html
+++ b/src/main/webapp/alluxio-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -262,7 +271,7 @@
<h3>
<a id="running-example" class="anchor" href="#running-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running Example</h3>
<ul>
-<li>Please refer to <a href="https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/AlluxioExample.scala" target=_blank>AlluxioExample</a>
+<li>Please refer to <a href="https://github.com/apache/carbondata/blob/master/examples/spark/src/main/scala/org/apache/carbondata/examples/AlluxioExample.scala" target=_blank>AlluxioExample</a>
</li>
</ul>
<h2>
@@ -290,7 +299,7 @@
<ul>
<li>Running the command in spark path</li>
</ul>
-<pre lang="$command"><code>./bin/spark-shell --jars ${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar,${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar
+<pre lang="$command"><code>./bin/spark-shell --jars ${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-2.0.0-SNAPSHOT-bin-spark2.3.4-hadoop2.7.2.jar,${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar
</code></pre>
<ul>
<li>Testing use alluxio by CarbonSession</li>
@@ -300,7 +309,7 @@ import org.apache.spark.sql.SparkSession
val carbon = SparkSession.builder().master("local").appName("test").getOrCreateCarbonSession("alluxio://localhost:19998/carbondata");
carbon.sql("CREATE TABLE carbon_alluxio(id String,name String, city String,age Int) STORED as carbondata");
-carbon.sql(s"LOAD DATA LOCAL INPATH '${CARBONDATA_PATH}/integration/spark-common-test/src/test/resources/sample.csv' into table carbon_alluxio");
+carbon.sql(s"LOAD DATA LOCAL INPATH '${CARBONDATA_PATH}/integration/spark/src/test/resources/sample.csv' into table carbon_alluxio");
carbon.sql("select * from carbon_alluxio").show
</code></pre>
<ul>
@@ -348,9 +357,9 @@ carbon.sql("select * from carbon_alluxio").show
<a id="command" class="anchor" href="#command" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Command</h4>
<pre lang="$command"><code>./bin/spark-submit \
--master local \
---jars ${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar,${CARBONDATA_PATH}/examples/spark2/target/carbondata-examples-1.6.0-SNAPSHOT.jar \
+--jars ${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar,${CARBONDATA_PATH}/examples/spark/target/carbondata-examples-2.0.0-SNAPSHOT.jar \
--class org.apache.carbondata.examples.AlluxioExample \
-${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar \
+${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-2.0.0-SNAPSHOT-bin-spark2.3.4-hadoop2.7.2.jar \
false
</code></pre>
<p><strong>NOTE</strong>: Please set runShell as false, which can avoid dependency on alluxio shell module.</p>
diff --git a/src/main/webapp/bloomfilter-datamap-guide.html b/src/main/webapp/bloomfilter-index-guide.html
similarity index 74%
copy from src/main/webapp/bloomfilter-datamap-guide.html
copy to src/main/webapp/bloomfilter-index-guide.html
index 2231c50..2ff3289 100644
--- a/src/main/webapp/bloomfilter-datamap-guide.html
+++ b/src/main/webapp/bloomfilter-index-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download <span class="caret"></span></a>
<ul class="dropdown-menu">
<li>
+ <a href="https://dist.apache.org/repos/dist/release/carbondata/2.0.0/"
+ target="_blank">Apache CarbonData 2.0.0</a></li>
+ <li>
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"
target="_blank">Apache CarbonData 1.6.1</a></li>
<li>
@@ -73,9 +76,6 @@
<a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"
target="_blank">Apache CarbonData 1.5.0</a></li>
<li>
- <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"
- target="_blank">Apache CarbonData 1.4.1</a></li>
- <li>
<a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"
target="_blank">Release Archive</a></li>
</ul>
@@ -184,12 +184,11 @@
</div>
<div class="nav__item nav__item__with__subs">
- <a class="b-nav__datamap nav__item nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
- <a class="nav__item nav__sub__item" href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
- <a class="nav__item nav__sub__item" href="./lucene-datamap-guide.html">Lucene</a>
- <a class="nav__item nav__sub__item" href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
- <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a>
- <a class="nav__item nav__sub__item" href="./mv-datamap-guide.html">MV</a>
+ <a class="b-nav__datamap nav__item nav__sub__anchor" href="./index-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item" href="./bloomfilter-index-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item" href="./lucene-index-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item" href="./secondary-index-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item" href="../spatial-index-guide.html">Spatial Index</a>
</div>
<div class="nav__item nav__item__with__subs">
@@ -199,8 +198,13 @@
</div>
<a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__mv nav__item" href="./mv-guide.html">MV</a>
<a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a>
<a class="b-nav__indexserver nav__item" href="./index-server.html">Index Server</a>
+ <a class="b-nav__prestodb nav__item" href="./prestodb-guide.html">PrestoDB Integration</a>
+ <a class="b-nav__prestosql nav__item" href="./prestosql-guide.html">PrestoSQL Integration</a>
+ <a class="b-nav__flink nav__item" href="./flink-integration-guide.html">Flink Integration</a>
+ <a class="b-nav__scd nav__item" href="./scd-and-cdc-guide.html">SCD & CDC</a>
<a class="b-nav__faq nav__item" href="./faq.html">FAQ</a>
<a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
<a class="b-nav__security nav__item" href="./security.html">Security</a>
@@ -215,8 +219,13 @@
<div class="b-nav__datamap navindicator__item"></div>
<div class="b-nav__api navindicator__item"></div>
<div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__mv navindicator__item"></div>
<div class="b-nav__s3 navindicator__item"></div>
<div class="b-nav__indexserver navindicator__item"></div>
+ <div class="b-nav__prestodb navindicator__item"></div>
+ <div class="b-nav__prestosql navindicator__item"></div>
+ <div class="b-nav__flink navindicator__item"></div>
+ <div class="b-nav__scd navindicator__item"></div>
<div class="b-nav__faq navindicator__item"></div>
<div class="b-nav__contri navindicator__item"></div>
<div class="b-nav__security navindicator__item"></div>
@@ -231,50 +240,53 @@
<div class="col-sm-12 col-md-12">
<div>
<h1>
-<a id="carbondata-bloomfilter-datamap" class="anchor" href="#carbondata-bloomfilter-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter DataMap</h1>
+<a id="carbondata-bloomfilter-index" class="anchor" href="#carbondata-bloomfilter-index" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData BloomFilter Index</h1>
<ul>
-<li><a href="#datamap-management">DataMap Management</a></li>
-<li><a href="#bloomfilter-datamap-introduction">BloomFilter Datamap Introduction</a></li>
+<li><a href="#index-management">Index Management</a></li>
+<li><a href="#bloomfilter-index-introduction">BloomFilter Index Introduction</a></li>
<li><a href="#loading-data">Loading Data</a></li>
<li><a href="#querying-data">Querying Data</a></li>
-<li><a href="#data-management-with-bloomfilter-datamap">Data Management</a></li>
+<li><a href="#data-management-with-bloomfilter-index">Data Management</a></li>
<li><a href="#useful-tips">Useful Tips</a></li>
</ul>
<h4>
-<a id="datamap-management" class="anchor" href="#datamap-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DataMap Management</h4>
-<p>Creating BloomFilter DataMap</p>
-<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
-ON TABLE main_table
-USING 'bloomfilter'
-DMPROPERTIES ('index_columns'='city, name', 'BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
+<a id="index-management" class="anchor" href="#index-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Index Management</h4>
+<p>Creating BloomFilter Index</p>
+<pre><code>CREATE INDEX [IF NOT EXISTS] index_name
+ON TABLE main_table (city,name)
+AS 'bloomfilter'
+PROPERTIES ('BLOOM_SIZE'='640000', 'BLOOM_FPP'='0.00001')
</code></pre>
-<p>Dropping Specified DataMap</p>
-<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
-ON TABLE main_table
+<p>Dropping Specified Index</p>
+<pre><code>DROP INDEX [IF EXISTS] index_name
+ON [TABLE] main_table
</code></pre>
-<p>Showing all DataMaps on this table</p>
-<pre><code>SHOW DATAMAP
-ON TABLE main_table
+<p>Showing all Indexes on this table</p>
+<pre><code>SHOW INDEXES
+ON [TABLE] main_table
</code></pre>
-<p>Disable DataMap</p>
<blockquote>
-<p>The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
+<p>NOTE: Keywords given inside <code>[]</code> is optional.</p>
+</blockquote>
+<p>Disable Index</p>
+<blockquote>
+<p>The index by default is enabled. To support tuning on query, we can disable a specific index during query to observe whether we can gain performance enhancement from it. This is effective only for current session.</p>
</blockquote>
-<pre><code>// disable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = false
-// enable the datamap
-SET carbon.datamap.visible.dbName.tableName.dataMapName = true
+<pre><code>// disable the index
+SET carbon.index.visible.dbName.tableName.indexName = false
+// enable the index
+SET carbon.index.visible.dbName.tableName.indexName = true
</code></pre>
<h2>
-<a id="bloomfilter-datamap-introduction" class="anchor" href="#bloomfilter-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter DataMap Introduction</h2>
+<a id="bloomfilter-index-introduction" class="anchor" href="#bloomfilter-index-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter Index Introduction</h2>
<p>A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.
-Carbondata introduced BloomFilter as an index datamap to enhance the performance of querying with precise value.
-It is well suitable for queries that do precise match on high cardinality columns(such as Name/ID).
+Carbondata introduced BloomFilter as an index to enhance the performance of querying with precise value.
+It is well suitable for queries that do precise matching on high cardinality columns(such as Name/ID).
Internally, CarbonData maintains a BloomFilter per blocklet for each index column to indicate that whether a value of the column is in this blocklet.
-Just like the other datamaps, BloomFilter datamap is managed along with main tables by CarbonData.
-User can create BloomFilter datamap on specified columns with specified BloomFilter configurations such as size and probability.</p>
-<p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p>
-<pre><code>CREATE TABLE datamap_test (
+Just like the other indexes, BloomFilter index is managed along with main tables by CarbonData.
+User can create BloomFilter index on specified columns with specified BloomFilter configurations such as size and probability.</p>
+<p>For instance, main table called <strong>index_test</strong> which is defined as:</p>
+<pre><code>CREATE TABLE index_test (
id string,
name string,
age int,
... 13454 lines suppressed ...