You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2020/02/04 19:41:43 UTC
[drill] branch gh-pages updated (57f2033 -> fd5041d)
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a change to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git.
from 57f2033 update readme.md
new fd42f80 Add docs for Drill Metastore
new b0836ef Changes after code review
new fd5041d More changes after code review
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_data/docs.json | 396 +++++++++++++++++---
_docs/getting-started/010-drill-introduction.md | 4 +-
_docs/performance-tuning/005-drill-metastore.md | 5 +
.../drill-metastore/010-using-drill-metastore.md | 408 +++++++++++++++++++++
.../drill-metastore/030-drill-iceberg-metastore.md | 69 ++++
.../060-querying-the-information-schema.md | 9 +-
_docs/rn/005-1.16.0-rn.md | 4 +-
.../sql-commands/005-supported-sql-commands.md | 46 +--
.../007-analyze-table-refresh-metadata.md | 112 ++++++
....md => 009-analyze-table-compute-statistics.md} | 81 ++--
10 files changed, 1013 insertions(+), 121 deletions(-)
create mode 100644 _docs/performance-tuning/005-drill-metastore.md
create mode 100644 _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
create mode 100644 _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
create mode 100644 _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
rename _docs/sql-reference/sql-commands/{009-analyze-table.md => 009-analyze-table-compute-statistics.md} (75%)
[drill] 03/03: More changes after code review
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git
commit fd5041dab391c4d5c66d3bf6b442fb8f901d4217
Author: Volodymyr Vysotskyi <vv...@gmail.com>
AuthorDate: Fri Jan 31 18:33:43 2020 +0200
More changes after code review
---
_data/docs.json | 308 ++++++++++-----------
.../drill-metastore/010-using-drill-metastore.md | 100 ++++---
.../drill-metastore/030-drill-iceberg-metastore.md | 14 +-
3 files changed, 229 insertions(+), 193 deletions(-)
diff --git a/_data/docs.json b/_data/docs.json
index b62691b..0b74254 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -3171,8 +3171,8 @@
}
],
"children": [],
- "next_title": "Drill Metastore",
- "next_url": "/docs/drill-metastore/",
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -4572,12 +4572,12 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Performance Tuning Introduction",
+ "next_url": "/docs/performance-tuning-introduction/",
"parent": "Drill Metastore",
"previous_title": "Using Drill Metastore",
"previous_url": "/docs/using-drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "relative_path": "_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md",
"title": "Drill Iceberg Metastore",
"url": "/docs/drill-iceberg-metastore/"
},
@@ -4639,12 +4639,12 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Performance Tuning Introduction",
+ "next_url": "/docs/performance-tuning-introduction/",
"parent": "Drill Metastore",
"previous_title": "Using Drill Metastore",
"previous_url": "/docs/using-drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "relative_path": "_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md",
"title": "Drill Iceberg Metastore",
"url": "/docs/drill-iceberg-metastore/"
}
@@ -4652,9 +4652,9 @@
"next_title": "Using Drill Metastore",
"next_url": "/docs/using-drill-metastore/",
"parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
- "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
+ "previous_title": "Performance Tuning",
+ "previous_url": "/docs/performance-tuning/",
+ "relative_path": "_docs/performance-tuning/005-drill-metastore.md",
"title": "Drill Metastore",
"url": "/docs/drill-metastore/"
},
@@ -5559,8 +5559,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Drill Iceberg Metastore",
- "previous_url": "/docs/drill-iceberg-metastore/",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
@@ -8185,12 +8185,72 @@
"url": "/docs/performance-tuning/"
}
],
+ "children": [
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Performance Tuning Introduction",
+ "next_url": "/docs/performance-tuning-introduction/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ }
+ ],
+ "next_title": "Using Drill Metastore",
+ "next_url": "/docs/using-drill-metastore/",
+ "parent": "Performance Tuning",
+ "previous_title": "Performance Tuning",
+ "previous_url": "/docs/performance-tuning/",
+ "relative_path": "_docs/performance-tuning/005-drill-metastore.md",
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
"children": [],
"next_title": "Partition Pruning",
"next_url": "/docs/partition-pruning/",
"parent": "Performance Tuning",
- "previous_title": "Performance Tuning",
- "previous_url": "/docs/performance-tuning/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/010-performance-tuning-introduction.md",
"title": "Performance Tuning Introduction",
"url": "/docs/performance-tuning-introduction/"
@@ -8507,8 +8567,8 @@
}
],
"children": [],
- "next_title": "Drill Metastore",
- "next_url": "/docs/drill-metastore/",
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -8537,66 +8597,6 @@
{
"breadcrumbs": [
{
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [],
- "next_title": "Drill Iceberg Metastore",
- "next_url": "/docs/drill-iceberg-metastore/",
- "parent": "Drill Metastore",
- "previous_title": "Drill Metastore",
- "previous_url": "/docs/drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
- "title": "Using Drill Metastore",
- "url": "/docs/using-drill-metastore/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
- "parent": "Drill Metastore",
- "previous_title": "Using Drill Metastore",
- "previous_url": "/docs/using-drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
- "title": "Drill Iceberg Metastore",
- "url": "/docs/drill-iceberg-metastore/"
- }
- ],
- "next_title": "Using Drill Metastore",
- "next_url": "/docs/using-drill-metastore/",
- "parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
- "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [
- {
- "breadcrumbs": [
- {
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
},
@@ -8640,8 +8640,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Drill Iceberg Metastore",
- "previous_url": "/docs/drill-iceberg-metastore/",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
@@ -8724,8 +8724,8 @@
"url": "/docs/monitoring-metrics/"
}
],
- "next_title": "Performance Tuning Introduction",
- "next_url": "/docs/performance-tuning-introduction/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "",
"previous_title": "Monitoring and Canceling Queries in the Drill Web UI",
"previous_url": "/docs/monitoring-and-canceling-queries-in-the-drill-web-ui/",
@@ -8744,8 +8744,8 @@
"next_title": "Partition Pruning",
"next_url": "/docs/partition-pruning/",
"parent": "Performance Tuning",
- "previous_title": "Performance Tuning",
- "previous_url": "/docs/performance-tuning/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/010-performance-tuning-introduction.md",
"title": "Performance Tuning Introduction",
"url": "/docs/performance-tuning-introduction/"
@@ -9793,8 +9793,8 @@
}
],
"children": [],
- "next_title": "Drill Metastore",
- "next_url": "/docs/drill-metastore/",
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -19675,12 +19675,72 @@
"url": "/docs/performance-tuning/"
}
],
+ "children": [
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Performance Tuning Introduction",
+ "next_url": "/docs/performance-tuning-introduction/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ }
+ ],
+ "next_title": "Using Drill Metastore",
+ "next_url": "/docs/using-drill-metastore/",
+ "parent": "Performance Tuning",
+ "previous_title": "Performance Tuning",
+ "previous_url": "/docs/performance-tuning/",
+ "relative_path": "_docs/performance-tuning/005-drill-metastore.md",
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
"children": [],
"next_title": "Partition Pruning",
"next_url": "/docs/partition-pruning/",
"parent": "Performance Tuning",
- "previous_title": "Performance Tuning",
- "previous_url": "/docs/performance-tuning/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/010-performance-tuning-introduction.md",
"title": "Performance Tuning Introduction",
"url": "/docs/performance-tuning-introduction/"
@@ -19997,8 +20057,8 @@
}
],
"children": [],
- "next_title": "Drill Metastore",
- "next_url": "/docs/drill-metastore/",
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -20027,66 +20087,6 @@
{
"breadcrumbs": [
{
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [],
- "next_title": "Drill Iceberg Metastore",
- "next_url": "/docs/drill-iceberg-metastore/",
- "parent": "Drill Metastore",
- "previous_title": "Drill Metastore",
- "previous_url": "/docs/drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
- "title": "Using Drill Metastore",
- "url": "/docs/using-drill-metastore/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
- "parent": "Drill Metastore",
- "previous_title": "Using Drill Metastore",
- "previous_url": "/docs/using-drill-metastore/",
- "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
- "title": "Drill Iceberg Metastore",
- "url": "/docs/drill-iceberg-metastore/"
- }
- ],
- "next_title": "Using Drill Metastore",
- "next_url": "/docs/using-drill-metastore/",
- "parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
- "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
- "title": "Drill Metastore",
- "url": "/docs/drill-metastore/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Performance Tuning",
- "url": "/docs/performance-tuning/"
- }
- ],
- "children": [
- {
- "breadcrumbs": [
- {
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
},
@@ -20130,8 +20130,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Drill Iceberg Metastore",
- "previous_url": "/docs/drill-iceberg-metastore/",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
@@ -20214,8 +20214,8 @@
"url": "/docs/monitoring-metrics/"
}
],
- "next_title": "Performance Tuning Introduction",
- "next_url": "/docs/performance-tuning-introduction/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "",
"previous_title": "Monitoring and Canceling Queries in the Drill Web UI",
"previous_url": "/docs/monitoring-and-canceling-queries-in-the-drill-web-ui/",
@@ -23026,4 +23026,4 @@
"url": "/docs/ecosystem/"
}
]
-}
+}
\ No newline at end of file
diff --git a/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md b/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
index d1dceb7..42f93c1 100644
--- a/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
+++ b/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
@@ -1,12 +1,12 @@
---
title: "Using Drill Metastore"
parent: "Drill Metastore"
-date: 2020-01-30
+date: 2020-01-31
---
Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
-The Metastore is an Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
@@ -27,12 +27,10 @@ Once you enable the Metastore, the next step is to populate it with data. Drill
query performance. In general, large tables benefit from statistics more than small tables do.
Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
- and computes some metadata like MIN / MAX column values and NULLS COUNT designated as "metadata" to be able to
- produce more optimizations like filter push-down, etc. If `planner.statistics.use` option is enabled, this command
- will also calculate and store table statistics into Drill Metastore.
-
-Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
- in the same way as it is done during regular select.
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
## Configuration
@@ -42,13 +40,24 @@ indicated in `drill-metastore-distrib.conf`.
All configuration properties should reside in `drill.metastore` namespace.
Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
### Metastore Components
-Metastore can store metadata for various components: tables, views etc.
+Metastore can store metadata for various components: tables, views, etc.
Current implementation provides fully functioning support for tables component.
Views component support is not implemented but contains stub methods to show
-how new Metastore components like udfs, storage plugins, etc. be added in future.
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
### Metastore Tables
@@ -56,15 +65,15 @@ Metastore Tables component contains metadata about Drill tables, including gener
information about table segments, files, row groups, partitions.
Full table metadata consists of two major concepts: general information and top-level segments metadata.
-Table general information contains basic table information and corresponds to `BaseTableMetadata` class.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
-Table can be non-partitioned and partitioned. Non-partitioned tables, have only one top-level segment
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment
which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
-Each top-level segment can include metadata about inner segments, files, row groups and partitions.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
-Unique table identifier in Metastore Tables is combination of storage plugin, workspace and table name.
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
-is storage plugin, workspace, table name and metadata key.
+is storage plugin, workspace, table name, and metadata key.
### Related Session/System Options
@@ -74,7 +83,7 @@ The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or
Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
- **metastore.metadata.store.depth_level**
-Specifies maximum level depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
- **metastore.retrieval.retry_attempts**
If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
@@ -91,19 +100,32 @@ Enable `planner.statistics.use` to be able to use statistics during query planni
- **metastore.metadata.ctas.auto-collect**
Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
This option causes Drill to gather schema and statistics for those tables automatically as they are written.
-This option is not active for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
- **drill.exec.storage.implicit.last_modified_time.column.label**
-Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze.
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
- **drill.exec.storage.implicit.row_group_index.column.label**
-Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze.
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
- **drill.exec.storage.implicit.row_group_length.column.label**
-Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze.
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
- **drill.exec.storage.implicit.row_group_start.column.label**
-Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze.
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
## Incremental analysis
-If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update statistics, called "incremental analysis."
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
actual metadata from the Metastore where possible.
@@ -116,9 +138,9 @@ If either of these two conditions is false, Drill will perform a full analysis o
## General Information
- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
-- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire tables (all files.)
-When you next issue the same command, Drill will scan only those files added since the previous run.
-The command will return the message if table statistics are up-to-date:
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
```
@@ -133,24 +155,24 @@ apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
### Metadata usage
Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
-Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups which
-do not contain data which your query needs. If you add new files or directories, and do not rerun `ANALYZE TABLE`,
-then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so that
- Drill can use table metadata when possible.
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
### Limitations
-This feature is currently in the alpha phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
## Examples
-Examples throughout this topic use the files and directories described in the following section, Directory and File Setup.
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
### Directory and File Setup
-Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and unpack
- archive.
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
@@ -172,7 +194,7 @@ SELECT count(*) FROM dfs.tmp.lineitem;
1 row selected (0.291 seconds)
```
-Notice that the query plan contains group scan with `usedMetastore = false`:
+Notice that the query plan contains a group scan with `usedMetastore = false`:
```
@@ -183,6 +205,12 @@ Notice that the query plan contains group scan with `usedMetastore = false`:
### Computing and storing table metadata to Drill Metastore
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
be computed and stored into the Drill Metastore:
@@ -199,7 +227,7 @@ apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
The output of this command provides the status of the command execution and its summary.
-Once, its metadata is collected and stored, it will be used when querying the table. To ensure that it was used, its
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
@@ -227,6 +255,8 @@ apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
Description of Metastore-specific columns:
diff --git a/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md b/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
index 1294286..744569b 100644
--- a/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
+++ b/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
@@ -1,12 +1,18 @@
---
title: "Drill Iceberg Metastore"
parent: "Drill Metastore"
-date:
+date: 2020-01-31
---
-Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
- details on how to configure Iceberg Metastore implementation and its option descriptions, please refer to
- [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For Drill 1.17,
+ this is default Drill Metastore implementation. For details on how to configure Iceberg Metastore implementation and
+ its option descriptions, please refer to [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
+
+{% include startnote.html %}
+Iceberg table supports concurrent writes and transactions but they are only effective on file systems that support
+ atomic rename.
+If the file system does not support atomic rename, it could lead to inconsistencies during concurrent writes.
+{% include endnote.html %}
### Iceberg Tables Location
[drill] 01/03: Add docs for Drill Metastore
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git
commit fd42f80df2fab5a1a23a88ca96e3d59026091584
Author: Volodymyr Vysotskyi <vv...@gmail.com>
AuthorDate: Mon Jan 13 16:43:10 2020 +0200
Add docs for Drill Metastore
---
_data/docs.json | 156 +++++++++++++++-----
_docs/getting-started/010-drill-introduction.md | 4 +-
_docs/rn/005-1.16.0-rn.md | 4 +-
.../007-analyze-table-refresh-metadata.md | 158 +++++++++++++++++++++
....md => 009-analyze-table-compute-statistics.md} | 81 +++++------
5 files changed, 315 insertions(+), 88 deletions(-)
diff --git a/_data/docs.json b/_data/docs.json
index b8f927f..f74ef35 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -21,7 +21,7 @@
"title": "ALTER SYSTEM",
"url": "/docs/alter-system/"
},
- "ANALYZE TABLE": {
+ "ANALYZE TABLE COMPUTE STATISTICS": {
"breadcrumbs": [
{
"title": "SQL Commands",
@@ -36,11 +36,32 @@
"next_title": "SET",
"next_url": "/docs/set/",
"parent": "SQL Commands",
+ "previous_title": "ANALYZE TABLE REFRESH METADATA",
+ "previous_url": "/docs/analyze-table-refresh-metadata/",
+ "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md",
+ "title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "url": "/docs/analyze-table-compute-statistics/"
+ },
+ "ANALYZE TABLE REFRESH METADATA": {
+ "breadcrumbs": [
+ {
+ "title": "SQL Commands",
+ "url": "/docs/sql-commands/"
+ },
+ {
+ "title": "SQL Reference",
+ "url": "/docs/sql-reference/"
+ }
+ ],
+ "children": [],
+ "next_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "next_url": "/docs/analyze-table-compute-statistics/",
+ "parent": "SQL Commands",
"previous_title": "Supported SQL Commands",
"previous_url": "/docs/supported-sql-commands/",
- "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table.md",
- "title": "ANALYZE TABLE",
- "url": "/docs/analyze-table/"
+ "relative_path": "_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md",
+ "title": "ANALYZE TABLE REFRESH METADATA",
+ "url": "/docs/analyze-table-refresh-metadata/"
},
"About SQL Function Examples": {
"breadcrumbs": [
@@ -11315,8 +11336,8 @@
"next_title": "REFRESH TABLE METADATA",
"next_url": "/docs/refresh-table-metadata/",
"parent": "SQL Commands",
- "previous_title": "ANALYZE TABLE",
- "previous_url": "/docs/analyze-table/",
+ "previous_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "previous_url": "/docs/analyze-table-compute-statistics/",
"relative_path": "_docs/sql-reference/sql-commands/010-set.md",
"title": "SET",
"url": "/docs/set/"
@@ -11404,8 +11425,8 @@
}
],
"children": [],
- "next_title": "ANALYZE TABLE",
- "next_url": "/docs/analyze-table/",
+ "next_title": "ANALYZE TABLE REFRESH METADATA",
+ "next_url": "/docs/analyze-table-refresh-metadata/",
"parent": "SQL Commands",
"previous_title": "SQL Commands",
"previous_url": "/docs/sql-commands/",
@@ -11425,14 +11446,35 @@
}
],
"children": [],
- "next_title": "SET",
- "next_url": "/docs/set/",
+ "next_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "next_url": "/docs/analyze-table-compute-statistics/",
"parent": "SQL Commands",
"previous_title": "Supported SQL Commands",
"previous_url": "/docs/supported-sql-commands/",
- "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table.md",
- "title": "ANALYZE TABLE",
- "url": "/docs/analyze-table/"
+ "relative_path": "_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md",
+ "title": "ANALYZE TABLE REFRESH METADATA",
+ "url": "/docs/analyze-table-refresh-metadata/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "SQL Commands",
+ "url": "/docs/sql-commands/"
+ },
+ {
+ "title": "SQL Reference",
+ "url": "/docs/sql-reference/"
+ }
+ ],
+ "children": [],
+ "next_title": "SET",
+ "next_url": "/docs/set/",
+ "parent": "SQL Commands",
+ "previous_title": "ANALYZE TABLE REFRESH METADATA",
+ "previous_url": "/docs/analyze-table-refresh-metadata/",
+ "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md",
+ "title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "url": "/docs/analyze-table-compute-statistics/"
},
{
"breadcrumbs": [
@@ -11449,8 +11491,8 @@
"next_title": "REFRESH TABLE METADATA",
"next_url": "/docs/refresh-table-metadata/",
"parent": "SQL Commands",
- "previous_title": "ANALYZE TABLE",
- "previous_url": "/docs/analyze-table/",
+ "previous_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "previous_url": "/docs/analyze-table-compute-statistics/",
"relative_path": "_docs/sql-reference/sql-commands/010-set.md",
"title": "SET",
"url": "/docs/set/"
@@ -13067,8 +13109,8 @@
}
],
"children": [],
- "next_title": "ANALYZE TABLE",
- "next_url": "/docs/analyze-table/",
+ "next_title": "ANALYZE TABLE REFRESH METADATA",
+ "next_url": "/docs/analyze-table-refresh-metadata/",
"parent": "SQL Commands",
"previous_title": "SQL Commands",
"previous_url": "/docs/sql-commands/",
@@ -13088,14 +13130,35 @@
}
],
"children": [],
- "next_title": "SET",
- "next_url": "/docs/set/",
+ "next_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "next_url": "/docs/analyze-table-compute-statistics/",
"parent": "SQL Commands",
"previous_title": "Supported SQL Commands",
"previous_url": "/docs/supported-sql-commands/",
- "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table.md",
- "title": "ANALYZE TABLE",
- "url": "/docs/analyze-table/"
+ "relative_path": "_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md",
+ "title": "ANALYZE TABLE REFRESH METADATA",
+ "url": "/docs/analyze-table-refresh-metadata/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "SQL Commands",
+ "url": "/docs/sql-commands/"
+ },
+ {
+ "title": "SQL Reference",
+ "url": "/docs/sql-reference/"
+ }
+ ],
+ "children": [],
+ "next_title": "SET",
+ "next_url": "/docs/set/",
+ "parent": "SQL Commands",
+ "previous_title": "ANALYZE TABLE REFRESH METADATA",
+ "previous_url": "/docs/analyze-table-refresh-metadata/",
+ "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md",
+ "title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "url": "/docs/analyze-table-compute-statistics/"
},
{
"breadcrumbs": [
@@ -13112,8 +13175,8 @@
"next_title": "REFRESH TABLE METADATA",
"next_url": "/docs/refresh-table-metadata/",
"parent": "SQL Commands",
- "previous_title": "ANALYZE TABLE",
- "previous_url": "/docs/analyze-table/",
+ "previous_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "previous_url": "/docs/analyze-table-compute-statistics/",
"relative_path": "_docs/sql-reference/sql-commands/010-set.md",
"title": "SET",
"url": "/docs/set/"
@@ -14900,8 +14963,8 @@
}
],
"children": [],
- "next_title": "ANALYZE TABLE",
- "next_url": "/docs/analyze-table/",
+ "next_title": "ANALYZE TABLE REFRESH METADATA",
+ "next_url": "/docs/analyze-table-refresh-metadata/",
"parent": "SQL Commands",
"previous_title": "SQL Commands",
"previous_url": "/docs/sql-commands/",
@@ -20703,8 +20766,8 @@
}
],
"children": [],
- "next_title": "ANALYZE TABLE",
- "next_url": "/docs/analyze-table/",
+ "next_title": "ANALYZE TABLE REFRESH METADATA",
+ "next_url": "/docs/analyze-table-refresh-metadata/",
"parent": "SQL Commands",
"previous_title": "SQL Commands",
"previous_url": "/docs/sql-commands/",
@@ -20724,14 +20787,35 @@
}
],
"children": [],
- "next_title": "SET",
- "next_url": "/docs/set/",
+ "next_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "next_url": "/docs/analyze-table-compute-statistics/",
"parent": "SQL Commands",
"previous_title": "Supported SQL Commands",
"previous_url": "/docs/supported-sql-commands/",
- "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table.md",
- "title": "ANALYZE TABLE",
- "url": "/docs/analyze-table/"
+ "relative_path": "_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md",
+ "title": "ANALYZE TABLE REFRESH METADATA",
+ "url": "/docs/analyze-table-refresh-metadata/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "SQL Commands",
+ "url": "/docs/sql-commands/"
+ },
+ {
+ "title": "SQL Reference",
+ "url": "/docs/sql-reference/"
+ }
+ ],
+ "children": [],
+ "next_title": "SET",
+ "next_url": "/docs/set/",
+ "parent": "SQL Commands",
+ "previous_title": "ANALYZE TABLE REFRESH METADATA",
+ "previous_url": "/docs/analyze-table-refresh-metadata/",
+ "relative_path": "_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md",
+ "title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "url": "/docs/analyze-table-compute-statistics/"
},
{
"breadcrumbs": [
@@ -20748,8 +20832,8 @@
"next_title": "REFRESH TABLE METADATA",
"next_url": "/docs/refresh-table-metadata/",
"parent": "SQL Commands",
- "previous_title": "ANALYZE TABLE",
- "previous_url": "/docs/analyze-table/",
+ "previous_title": "ANALYZE TABLE COMPUTE STATISTICS",
+ "previous_url": "/docs/analyze-table-compute-statistics/",
"relative_path": "_docs/sql-reference/sql-commands/010-set.md",
"title": "SET",
"url": "/docs/set/"
@@ -22720,4 +22804,4 @@
"url": "/docs/ecosystem/"
}
]
-}
\ No newline at end of file
+}
diff --git a/_docs/getting-started/010-drill-introduction.md b/_docs/getting-started/010-drill-introduction.md
index 7035a98..5ab61aa 100644
--- a/_docs/getting-started/010-drill-introduction.md
+++ b/_docs/getting-started/010-drill-introduction.md
@@ -1,6 +1,6 @@
---
title: "Drill Introduction"
-date: 2019-12-26
+date: 2020-01-13
parent: "Getting Started"
---
Drill is an Apache open-source SQL query engine for Big Data exploration.
@@ -46,7 +46,7 @@ Metastore support:
## What's New in Apache Drill 1.16
-- [ANALYZE TABLE statement]({{site.baseurl}}/docs/analyze-table/) to computes statistics on Parquet data ([DRILL-1328](https://issues.apache.org/jira/browse/DRILL-1328))
+- [ANALYZE TABLE statement]({{site.baseurl}}/docs/analyze-table-compute-statistics/) to computes statistics on Parquet data ([DRILL-1328](https://issues.apache.org/jira/browse/DRILL-1328))
- [CREATE OR REPLACE SCHEMA command]({{site.baseurl}}/docs/create-or-replace-schema/) to define a schema for text files ([DRILL-6964](https://issues.apache.org/jira/browse/DRILL-6964))
- [REFRESH TABLE METADATA command]({{site.baseurl}}/docs/refresh-table-metadata/) can generate metadata cache files for specific columns ([DRILL-7058](https://issues.apache.org/jira/browse/DRILL-7058))
- [SYSLOG (RFC-5424) Format Plugin]({{site.baseurl}}/docs/syslog-format-plugin/) ([DRILL-6582](https://issues.apache.org/jira/browse/DRILL-6582))
diff --git a/_docs/rn/005-1.16.0-rn.md b/_docs/rn/005-1.16.0-rn.md
index ac07ab7..88fabd7 100644
--- a/_docs/rn/005-1.16.0-rn.md
+++ b/_docs/rn/005-1.16.0-rn.md
@@ -12,7 +12,7 @@ Today, we're happy to announce the availability of Drill 1.16.0. You can downloa
This release of Drill provides the following new features and improvements:
-- [ANALYZE TABLE statement]({{site.baseurl}}/docs/analyze-table/) to computes statistics on Parquet data ([DRILL-1328](https://issues.apache.org/jira/browse/DRILL-1328))
+- [ANALYZE TABLE statement]({{site.baseurl}}/docs/analyze-table-compute-statistics/) to computes statistics on Parquet data ([DRILL-1328](https://issues.apache.org/jira/browse/DRILL-1328))
- [CREATE OR REPLACE SCHEMA command]({{site.baseurl}}/docs/create-or-replace-schema/) to define a schema for text files ([DRILL-6964](https://issues.apache.org/jira/browse/DRILL-6964))
- [REFRESH TABLE METADATA command]({{site.baseurl}}/docs/refresh-table-metadata/) can generate metadata cache files for specific columns ([DRILL-7058](https://issues.apache.org/jira/browse/DRILL-7058))
- [SYSLOG (RFC-5424) Format Plugin]({{site.baseurl}}/docs/syslog-format-plugin/) ([DRILL-6582](https://issues.apache.org/jira/browse/DRILL-6582))
@@ -498,4 +498,4 @@ The following sections provide a complete list of all the fixes and improvements
</li>
</ul>
-
\ No newline at end of file
+
diff --git a/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md b/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
new file mode 100644
index 0000000..21fa711
--- /dev/null
+++ b/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
@@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+ SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+ ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+ REFRESH METADATA ['level' LEVEL]
+ [{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+ [ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
+If statistics usage is disabled (`planner.enable_statistics` is set to `false`), an error will be thrown when this clause is specified.
+
+*ESTIMATE*
+Generates estimated statistics for the table to be stored into the Metastore. Currently is not supported.
+
+*(column1, column2, ...)*
+The name of the column(s) for which Drill will generate statistics.
+
+*SAMPLE*
+Optional. Indicates that compute statistics should run on a subset of the data.
+
+*number PERCENT*
+An integer that specifies the percentage of data on which to compute statistics. For example, if a table has 100 rows, `SAMPLE 50 PERCENT` indicates that statistics should be computed on 50 rows. The optimizer selects the rows at random.
+
+## Related Options
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies maximum level depth for collecting metadata. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+Specifies the number of attempts for retrying query planning after detecting that query metadata is changed.
+If the number of retries was exceeded, query will be planned without metadata information from the Metastore. Default is 5.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using file metadata cache for the case when required metadata is absent in the Metastore. Default is true.
+- **metastore.metadata.use_schema**
+Enables schema usage, stored to the Metastore. Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables statistics usage, stored in the Metastore, at the planning stage. Default is `true`.
+- **metastore.metadata.ctas.auto-collect**
+Specifies whether schema and / or column statistics will be automatically collected for every table after CTAS and CTTAS.
+This option is not active for now. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. For internal usage when producing Metastore analyze.
+
+## Related Commands
+
+To drop table metadata from the Metastore, the following command may be used:
+
+ ANALYZE TABLE [table_name] DROP [METADATA|STATISTICS] [IF EXISTS]
+
+It will not throw an exception for absent table metadata if `IF EXISTS` clause was specified.
+
+ ANALYZE TABLE [workspace.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT]
+
+See [ANALYZE TABLE COMPUTE STATISTICS]({{site.baseurl}}/docs/analyze-table-compute-statistics).
+
+## Usage Notes
+
+### General Information
+
+- Currently `ANALYZE TABLE REFRESH METADATA` statement can compute and store metadata only for Parquet tables within `dfs` storage plugins.
+- For the case when `ANALYZE TABLE REFRESH METADATA` command is executed for the first time, whole table metadata will be collected and stored into Metastore.
+If analyze was already executed for the table, and table data wasn't changed, all further analyze commands wouldn't trigger table analyzing and message that table metadata is up to date will be returned.
+
+### Incremental analyze
+
+For the case when some table data was updated, Drill will try to execute incremental analyze - calculate metadata only for updated data and reuse required metadata from the Metastore.
+
+Incremental analyze wouldn't be produced for the following cases:
+- list of interesting columns specified in analyze is not a subset of interesting columns from the previous analyze;
+- specified metadata level differs from the metadata level in previous analyze.
+
+### Metadata usage
+
+Drill provides the ability to use metadata obtained from the Metastore at the planning stage to prune segments, files
+ and row groups.
+
+Tables metadata from the Metastore is exposed to `INFORMATION_SCHEMA` tables (if Metastore usage is enabled).
+
+The following tables are populated with table metadata from the Metastore:
+
+`TABLES` table has the following additional columns populated from the Metastore:
+
+- `TABLE_SOURCE` - table data type: `PARQUET`, `CSV`, `JSON`
+- `LOCATION` - table location: `/tmp/nation`
+- `NUM_ROWS` - number of rows in a table if known, `null` if not known
+- `LAST_MODIFIED_TIME` - table's last modification time
+
+`COLUMNS` table has the following additional columns populated from the Metastore:
+
+- `COLUMN_DEFAULT` - column default value
+- `COLUMN_FORMAT` - usually applicable for date time columns: `yyyy-MM-dd`
+- `NUM_NULLS` - number of nulls in column values
+- `MIN_VAL` - column min value in String representation: `aaa`
+- `MAX_VAL` - column max value in String representation: `zzz`
+- `NDV` - number of distinct values in column, expressed in Double
+- `EST_NUM_NON_NULLS` - estimated number of non null values, expressed in Double
+- `IS_NESTED` - if column is nested. Nested columns are extracted from columns with struct type.
+
+`PARTITIONS` table has the following additional columns populated from the Metastore:
+
+- `TABLE_CATALOG` - table catalog (currently we have only one catalog): `DRILL`
+- `TABLE_SCHEMA` - table schema: `dfs.tmp`
+- `TABLE_NAME` - table name: `nation`
+- `METADATA_KEY` - top level segment key, the same for all nested segments and partitions: `part_int=3`
+- `METADATA_TYPE` - `SEGMENT` or `PARTITION`
+- `METADATA_IDENTIFIER` - current metadata identifier: `part_int=3/part_varchar=g`
+- `PARTITION_COLUMN` - partition column name: `part_varchar`
+- `PARTITION_VALUE` - partition column value: `g`
+- `LOCATION` - segment location, `null` for partitions: `/tmp/nation/part_int=3`
+- `LAST_MODIFIED_TIME` - last modification time
+
+## Limitations
+
+This feature is currently in the alpha phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
diff --git a/_docs/sql-reference/sql-commands/009-analyze-table.md b/_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md
similarity index 75%
rename from _docs/sql-reference/sql-commands/009-analyze-table.md
rename to _docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md
index 9251d0e..a642a40 100644
--- a/_docs/sql-reference/sql-commands/009-analyze-table.md
+++ b/_docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md
@@ -1,11 +1,11 @@
---
-title: "ANALYZE TABLE"
+title: "ANALYZE TABLE COMPUTE STATISTICS"
date: 2019-05-31
parent: "SQL Commands"
---
-Drill 1.16 and later supports the ANALYZE TABLE statement. The ANALYZE TABLE statement computes statistics on Parquet data stored in tables and directories. The optimizer in Drill uses statistics to estimate filter, aggregation, and join cardinalities and create an optimal query plan.
-ANALYZE TABLE writes statistics to a JSON file in the `.stats.drill` directory, for example `/user/table1/.stats.drill/0_0.json`.
+Drill 1.16 and later supports the ANALYZE TABLE COMPUTE STATISTICS statement. The ANALYZE TABLE COMPUTE STATISTICS statement computes statistics on Parquet data stored in tables and directories. The optimizer in Drill uses statistics to estimate filter, aggregation, and join cardinalities and create an optimal query plan.
+ANALYZE TABLE COMPUTE STATISTICS writes statistics to a JSON file in the `.stats.drill` directory, for example `/user/table1/.stats.drill/0_0.json`.
Drill will not use the statistics for query planning unless you enable the `planner.statistics.use` option, as shown:
@@ -15,7 +15,7 @@ Alternatively, you can enable the option in the Drill Web UI at `http://<drill-h
## Syntax
-The ANALYZE TABLE statement supports the following syntax:
+The ANALYZE TABLE COMPUTE STATISTICS statement supports the following syntax:
ANALYZE TABLE [workspace.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT]
@@ -42,7 +42,7 @@ An integer that specifies the percentage of data on which to compute statistics.
## Related Command
-If you drop a table that you have already run ANALYZE TABLE against, the statistics are automatically removed with the table:
+If you drop a table that you have already run ANALYZE TABLE COMPUTE STATISTICS against, the statistics are automatically removed with the table:
DROP TABLE [IF EXISTS] [workspace.]name
@@ -50,7 +50,7 @@ To remove statistics for a table you want to keep, you must remove the directory
DROP TABLE [IF EXISTS] [workspace.]name/.stats.drill
-If you have already issued the ANALYZE TABLE statement against specific columns, table, or directory, you must run the DROP TABLE statement with `/.stats.drill` before you can successfully run the ANALYZE TABLE statement against the data source again, for example:
+If you have already issued the ANALYZE TABLE COMPUTE STATISTICS statement against specific columns, table, or directory, you must run the DROP TABLE statement with `/.stats.drill` before you can successfully run the ANALYZE TABLE COMPUTE STATISTICS statement against the data source again, for example:
DROP TABLE `table_stats/Tpch0.01/parquet/customer/.stats.drill`;
@@ -60,30 +60,30 @@ Note that `/.stats.drill` is the directory to which the JSON file with statistic
## Usage Notes
-- The ANALYZE TABLE statement can compute statistics for Parquet data stored in tables, columns, and directories within dfs storage plugins only.
-- The user running the ANALYZE TABLE statement must have read and write permissions on the data source.
+- The ANALYZE TABLE COMPUTE STATISTICS statement can compute statistics for Parquet data stored in tables, columns, and directories within dfs storage plugins only.
+- The user running the ANALYZE TABLE COMPUTE STATISTICS statement must have read and write permissions on the data source.
- The optimizer in Drill computes the following types of statistics for each column:
- Rowcount (total number of entries in the table)
- Nonnullrowcount (total number of non-null entries in the table)
- NDV (total distinct values in the table)
- Avgwidth (average width, in bytes, of a column)
- Majortype (data type and data mode (OPTIONAL, REQUIRED, REPEATED) of the column values)
- - Histogram (represents the frequency distribution of values (numeric data) in a column) See [Histograms]({{site.baseurl}}/docs/analyze-table/#histograms).
+ - Histogram (represents the frequency distribution of values (numeric data) in a column) See [Histograms]({{site.baseurl}}/docs/analyze-table-compute-statistics/#histograms).
- When you look at the statistics file, statistics for each column display in the following format (c_nationkey is used as an example column):
{"column":"`c_nationkey`","majortype":{"type":"INT","mode":"REQUIRED"},"schema":1.0,"rowcount":1500.0,"nonnullrowcount":1500.0,"ndv":25,"avgwidth":4.0,"histogram":{"category":"numeric-equi-depth","numRowsPerBucket":150,"buckets":[0.0,2.0,4.0,7.0,9.0,12.0,15.199999999999978,17.0,19.0,22.0,24.0]}}
-- ANALYZE TABLE can compute statistics on nested scalar columns; however, you must explicitly state the columns, for example:
+- ANALYZE TABLE COMPUTE STATISTICS can compute statistics on nested scalar columns; however, you must explicitly state the columns, for example:
`ANALYZE TABLE employee_table COMPUTE STATISTICS (name.firstname, name.lastname);`
-- ANALYZE TABLE can compute statistics at the root directory level, but not at the partition level.
+- ANALYZE TABLE COMPUTE STATISTICS can compute statistics at the root directory level, but not at the partition level.
Drill does not compute statistics for complex types (maps, arrays).
## Related Options
-You can set the following options related to the ANALYZE TABLE statement at the system or session level with the SET (session level) or ALTER SYSTEM SET (system level) statements, or through the Drill Web UI at `http://<drill-hostname-or-ip>:8047/options`:
+You can set the following options related to the ANALYZE TABLE COMPUTE STATISTICS statement at the system or session level with the SET (session level) or ALTER SYSTEM SET (system level) statements, or through the Drill Web UI at `http://<drill-hostname-or-ip>:8047/options`:
- **planner.statistics.use**
-Enables the query planner to use statistics. When disabled, ANALYZE TABLE generates statistics, but the query planner will not use the statistics unless this option is enabled. Disabled (false) by default.
+Enables the query planner to use statistics. When disabled, ANALYZE TABLE COMPUTE STATISTICS generates statistics, but the query planner will not use the statistics unless this option is enabled. Disabled (false) by default.
- **exec.statistics.ndv_accuracy**
Controls the trade-off between NDV statistic computation memory cost and accuracy. Controls the amount of memory for estimates. More memory produces more accurate estimates. The default value should suffice for most scenarios. Default is 20. Range is 0- 30.
- **exec.statistics.ndv_extrapolation_bf_elements**
@@ -99,7 +99,7 @@ Controls the 'compression' factor for the TDigest algorithm used for histogram s
## Reserved Keywords
-The ANALYZE TABLE statement introduces the following reserved keywords:
+The ANALYZE TABLE COMPUTE STATISTICS statement introduces the following reserved keywords:
Analyze
Compute
@@ -112,11 +112,11 @@ If you use any of these words in a Drill query, you must enclose the word in bac
SELECT * FROM `estimate`;
-## ANALYZE TABLE Performance
+## ANALYZE TABLE COMPUTE STATISTICS Performance
-- After you run the ANALYZE TABLE statement, you can view the profile for ANALYZE in the Drill Web UI. Go to `http://<drill-hostname-or-ip>:8047/profiles`, and click the ANALYZE TABLE statement for which you want to view the profile.
+- After you run the ANALYZE TABLE COMPUTE STATISTICS statement, you can view the profile for ANALYZE in the Drill Web UI. Go to `http://<drill-hostname-or-ip>:8047/profiles`, and click the ANALYZE TABLE statement for which you want to view the profile.
- Should you notice any performance issues, you may want to decrease the value of the `planner.slice_target` option.
-- Generating statistics on large data sets can consume time and resources, such as memory and CPU. ANALYZE TABLE can compute statistics on a sample (subset of the data indicated as a percentage) to limit the amount of resources needed for computation. Drill still scans the entire data set, but only computes on the rows selected for sampling. Rows are randomly selected for the sample. Note that the quality of statistics increases with the sample size.
+- Generating statistics on large data sets can consume time and resources, such as memory and CPU. ANALYZE TABLE COMPUTE STATISTICS can compute statistics on a sample (subset of the data indicated as a percentage) to limit the amount of resources needed for computation. Drill still scans the entire data set, but only computes on the rows selected for sampling. Rows are randomly selected for the sample. Note that the quality of statistics increases with the sample size.
## Queries that Benefit from Statistics
Typically, the types of queries that benefit from statistics are those that include:
@@ -132,7 +132,7 @@ Typically, the types of queries that benefit from statistics are those that incl
Histograms show the distribution of data to determine if data is skewed or normally distributed. Histogram statistics improve the selectivity estimates used by the optimizer to create the most efficient query plans possible. Histogram statistics are useful for range predicates to help determine how many rows belong to a particular range.
-Running the ANALYZE TABLE statement generates equi-depth histogram statistics on each column in a table. Equi-depth histograms distribute distinct column values across buckets of varying widths, with all buckets having approximately the same number of rows. The fixed number of rows per bucket is predetermined by `ceil(number_rows/n)`, where `n` is the number of buckets. The number of distinct values in each bucket depends on the distribution of the values in a column. Equi-depth histogra [...]
+Running the ANALYZE TABLE COMPUTE STATISTICS statement generates equi-depth histogram statistics on each column in a table. Equi-depth histograms distribute distinct column values across buckets of varying widths, with all buckets having approximately the same number of rows. The fixed number of rows per bucket is predetermined by `ceil(number_rows/n)`, where `n` is the number of buckets. The number of distinct values in each bucket depends on the distribution of the values in a column. [...]
The following diagram shows the column values on the horizontal axis and the individual frequencies (dark blue) and total frequency of a bucket (light blue). In this example, the total number of rows = 64, hence the number of rows per bucket = `ceil(64/4) = 16`.
@@ -158,19 +158,19 @@ In this example, there are 10 buckets. Each bucket contains 150 rows, which is c
## Limitations
- Drill does not cache statistics.
-- ANALYZE TABLE runs only on directory-based Parquet tables.
-- ANALYZE TABLE cannot do the following:
+- ANALYZE TABLE COMPUTE STATISTICS runs only on directory-based Parquet tables.
+- ANALYZE TABLE COMPUTE STATISTICS cannot do the following:
- compute statistics on schema-less file formats, such as text and CSV
- provide up-to-date statistics for operational data due to potential mismatches that can occur between operational updates and manually running ANALYZE TABLE
-- Running the ANALYZE TABLE statement against multiple files in which some of the files have null values and others have no null values may return the following generic Drill error, which is not specific to the ANALYZE command:
+- Running the ANALYZE TABLE COMPUTE STATISTICS statement against multiple files in which some of the files have null values and others have no null values may return the following generic Drill error, which is not specific to the ANALYZE command:
Error: SYSTEM ERROR: IllegalStateException: Failure while reading vector.
Expected vector class of org.apache.drill.exec.vector.NullableBigIntVector
but was holding vector class org.apache.drill.exec.vector.IntVector, field= [`o_custkey` (INT:REQUIRED)]
- //If you encounter this error, run the ANALYZE TABLE statement on each file with null values individually instead of running the statement against all the files at once.
+ //If you encounter this error, run the ANALYZE TABLE COMPUTE STATISTICS statement on each file with null values individually instead of running the statement against all the files at once.
-- Running the ANALYZE TABLE statement creates the stats file, which changes the directory timestamp. The change of the timestamp automatically triggers the REFRESH TABLE METADATA command, even when the underlying data has not changed.
+- Running the ANALYZE TABLE COMPUTE STATISTICS statement creates the stats file, which changes the directory timestamp. The change of the timestamp automatically triggers the REFRESH TABLE METADATA command, even when the underlying data has not changed.
## EXAMPLES
@@ -201,7 +201,7 @@ The following query shows the columns and types of data in the “customer” ta
###Enabling Statistics for Query Planning
-You can run the ANALYZE TABLE statement at any time to compute statistics; however, you must enable the following option if you want Drill to use statistics during query planning:
+You can run the ANALYZE TABLE COMPUTE STATISTICS statement at any time to compute statistics; however, you must enable the following option if you want Drill to use statistics during query planning:
set `planner.statistics.use`=true;
+------+---------------------------------+
@@ -213,7 +213,7 @@ You can run the ANALYZE TABLE statement at any time to compute statistics; howev
###Computing Statistics
You can compute statistics on directories with Parquet data or on Parquet tables.
-You can run the ANALYZE TABLE statement on a subset of columns to generate statistics for those columns only, as shown:
+You can run the ANALYZE TABLE COMPUTE STATISTICS statement on a subset of columns to generate statistics for those columns only, as shown:
analyze table `table_stats/Tpch0.01/parquet/customer` compute statistics (c_custkey, c_nationkey, c_acctbal);
+----------+---------------------------+
@@ -222,7 +222,7 @@ You can run the ANALYZE TABLE statement on a subset of columns to generate stati
| 0_0 | 3 |
+----------+---------------------------+
-Or, you can run the ANALYZE TABLE statement on the entire table/directory if you want statistics generated for all the columns:
+Or, you can run the ANALYZE TABLE COMPUTE STATISTICS statement on the entire table/directory if you want statistics generated for all the columns:
analyze table `table_stats/Tpch0.01/parquet/customer` compute statistics;
+----------+---------------------------+
@@ -234,7 +234,7 @@ Or, you can run the ANALYZE TABLE statement on the entire table/directory if you
###Computing Statistics on a SAMPLE
-You can also run ANALYZE TABLE on a percentage of the data using the SAMPLE command, as shown:
+You can also run ANALYZE TABLE COMPUTE STATISTICS on a percentage of the data using the SAMPLE command, as shown:
ANALYZE TABLE `table_stats/Tpch0.01/parquet/customer` COMPUTE STATISTICS SAMPLE 50 PERCENT;
+----------+---------------------------+
@@ -247,9 +247,9 @@ You can also run ANALYZE TABLE on a percentage of the data using the SAMPLE comm
###Storing Statistics
When you generate statistics, a statistics directory (`.stats.drill`) is created with a JSON file that contains the statistical data.
-For tables, the `.stats.drill` directory is nested within the table directory. For example, if you ran ANALYZE TABLE against a table named “customer,” you could access the statistic file in `/customer/.stats.drill`. The JSON file is stored in the `.stats.drill` directory.
+For tables, the `.stats.drill` directory is nested within the table directory. For example, if you ran ANALYZE TABLE COMPUTE STATISTICS against a table named “customer,” you could access the statistic file in `/customer/.stats.drill`. The JSON file is stored in the `.stats.drill` directory.
-For directories, a new directory is written with the same name as the directory on which you ran ANALYZE TABLE, appended by `.stats.drill`. For example, if you ran ANALYZE TABLE against a directory named “customer,” you could access the JSON statistics file in the new `customer.stats.drill` directory.
+For directories, a new directory is written with the same name as the directory on which you ran ANALYZE TABLE COMPUTE STATISTICS, appended by `.stats.drill`. For example, if you ran ANALYZE TABLE COMPUTE STATISTICS against a directory named “customer,” you could access the JSON statistics file in the new `customer.stats.drill` directory.
You can query the statistics file to see the statistics generated for each column, as shown in the following two examples:
@@ -272,7 +272,7 @@ You can query the statistics file to see the statistics generated for each colum
###Dropping Statistics
-If you want to compute statistics on a table or directory that you have already run the ANALYZE TABLE statement against, you must first drop the statistics before you can run ANALYZE TABLE statement on the table again.
+If you want to compute statistics on a table or directory that you have already run the ANALYZE TABLE COMPUTE STATISTICS statement against, you must first drop the statistics before you can run ANALYZE TABLE statement on the table again.
The following example demonstrates how to drop statistics on a table:
@@ -303,9 +303,9 @@ When you drop statistics, the statistics directory no longer exists for the tabl
## Troubleshooting
-Typical errors you may get when running ANALYZE TABLE result from running the statement against an individual file or against a data source other than Parquet, as shown in the following examples:
+Typical errors you may get when running ANALYZE TABLE COMPUTE STATISTICS result from running the statement against an individual file or against a data source other than Parquet, as shown in the following examples:
-**Running ANALYZE TABLE on a file.**
+**Running ANALYZE TABLE COMPUTE STATISTICS on a file.**
ANALYZE TABLE `/parquet/nation.parquet` COMPUTE STATISTICS;
+--------+----------------------------------------------------------------------------------+
@@ -315,7 +315,7 @@ Typical errors you may get when running ANALYZE TABLE result from running the st
+--------+----------------------------------------------------------------------------------+
-**Running ANALYZE TABLE on a data source other than Parquet.**
+**Running ANALYZE TABLE COMPUTE STATISTICS on a data source other than Parquet.**
ANALYZE TABLE nation1_json COMPUTE STATISTICS;
+--------+----------------------------------------------------------------------------------+
@@ -323,18 +323,3 @@ Typical errors you may get when running ANALYZE TABLE result from running the st
+--------+----------------------------------------------------------------------------------+
| false | Table nation1_json is not supported by ANALYZE. Support is currently limited to directory-based Parquet tables. |
+--------+----------------------------------------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
[drill] 02/03: Changes after code review
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git
commit b0836ef4565ad83e352c88c6d42847c0d15c9bee
Author: Volodymyr Vysotskyi <vv...@gmail.com>
AuthorDate: Tue Jan 28 19:58:24 2020 +0200
Changes after code review
---
_data/docs.json | 250 +++++++++++++-
_docs/performance-tuning/005-drill-metastore.md | 5 +
.../drill-metastore/010-using-drill-metastore.md | 378 +++++++++++++++++++++
.../drill-metastore/030-drill-iceberg-metastore.md | 63 ++++
.../060-querying-the-information-schema.md | 9 +-
.../sql-commands/005-supported-sql-commands.md | 46 +--
.../007-analyze-table-refresh-metadata.md | 160 ++++-----
7 files changed, 770 insertions(+), 141 deletions(-)
diff --git a/_data/docs.json b/_data/docs.json
index f74ef35..b62691b 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -3171,8 +3171,8 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -4560,6 +4560,27 @@
"title": "Drill Explorer Introduction",
"url": "/docs/drill-explorer-introduction/"
},
+ "Drill Iceberg Metastore": {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ },
"Drill Introduction": {
"breadcrumbs": [
{
@@ -4577,6 +4598,66 @@
"title": "Drill Introduction",
"url": "/docs/drill-introduction/"
},
+ "Drill Metastore": {
+ "breadcrumbs": [
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ }
+ ],
+ "next_title": "Using Drill Metastore",
+ "next_url": "/docs/using-drill-metastore/",
+ "parent": "Performance Tuning",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
"Drill Plan Syntax": {
"breadcrumbs": [
{
@@ -5478,8 +5559,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
@@ -8426,8 +8507,8 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -8456,6 +8537,66 @@
{
"breadcrumbs": [
{
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ }
+ ],
+ "next_title": "Using Drill Metastore",
+ "next_url": "/docs/using-drill-metastore/",
+ "parent": "Performance Tuning",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [
+ {
+ "breadcrumbs": [
+ {
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
},
@@ -8499,8 +8640,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
@@ -9652,8 +9793,8 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -15682,6 +15823,27 @@
"title": "Using Drill Explorer",
"url": "/docs/using-drill-explorer/"
},
+ "Using Drill Metastore": {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
"Using Drill with BI Tools": {
"breadcrumbs": [
{
@@ -19835,8 +19997,8 @@
}
],
"children": [],
- "next_title": "Identifying Performance Issues",
- "next_url": "/docs/identifying-performance-issues/",
+ "next_title": "Drill Metastore",
+ "next_url": "/docs/drill-metastore/",
"parent": "Query Plans and Tuning",
"previous_title": "Throttling",
"previous_url": "/docs/throttling/",
@@ -19865,6 +20027,66 @@
{
"breadcrumbs": [
{
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Drill Iceberg Metastore",
+ "next_url": "/docs/drill-iceberg-metastore/",
+ "parent": "Drill Metastore",
+ "previous_title": "Drill Metastore",
+ "previous_url": "/docs/drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md",
+ "title": "Using Drill Metastore",
+ "url": "/docs/using-drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [],
+ "next_title": "Identifying Performance Issues",
+ "next_url": "/docs/identifying-performance-issues/",
+ "parent": "Drill Metastore",
+ "previous_title": "Using Drill Metastore",
+ "previous_url": "/docs/using-drill-metastore/",
+ "relative_path": "_docs/performance-tuning/drill-metastore/020-drill-iceberg-metastore.md",
+ "title": "Drill Iceberg Metastore",
+ "url": "/docs/drill-iceberg-metastore/"
+ }
+ ],
+ "next_title": "Using Drill Metastore",
+ "next_url": "/docs/using-drill-metastore/",
+ "parent": "Performance Tuning",
+ "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
+ "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "relative_path": "_docs/performance-tuning/050-drill-metastore.md",
+ "title": "Drill Metastore",
+ "url": "/docs/drill-metastore/"
+ },
+ {
+ "breadcrumbs": [
+ {
+ "title": "Performance Tuning",
+ "url": "/docs/performance-tuning/"
+ }
+ ],
+ "children": [
+ {
+ "breadcrumbs": [
+ {
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
},
@@ -19908,8 +20130,8 @@
"next_title": "Query Plans",
"next_url": "/docs/query-plans/",
"parent": "Performance Tuning",
- "previous_title": "Controlling Parallelization to Balance Performance with Multi-Tenancy",
- "previous_url": "/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/",
+ "previous_title": "Drill Iceberg Metastore",
+ "previous_url": "/docs/drill-iceberg-metastore/",
"relative_path": "_docs/performance-tuning/050-identifying-performance-issues.md",
"title": "Identifying Performance Issues",
"url": "/docs/identifying-performance-issues/"
diff --git a/_docs/performance-tuning/005-drill-metastore.md b/_docs/performance-tuning/005-drill-metastore.md
new file mode 100644
index 0000000..3ff1a64
--- /dev/null
+++ b/_docs/performance-tuning/005-drill-metastore.md
@@ -0,0 +1,5 @@
+---
+title: "Drill Metastore"
+parent: "Performance Tuning"
+date:
+---
diff --git a/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md b/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
new file mode 100644
index 0000000..d1dceb7
--- /dev/null
+++ b/_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
@@ -0,0 +1,378 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-30
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is an Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+ SET `metastore.enabled` = true;
+ ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ and computes some metadata like MIN / MAX column values and NULLS COUNT designated as "metadata" to be able to
+ produce more optimizations like filter push-down, etc. If `planner.statistics.use` option is enabled, this command
+ will also calculate and store table statistics into Drill Metastore.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ in the same way as it is done during regular select.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like udfs, storage plugins, etc. be added in future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to `BaseTableMetadata` class.
+
+Table can be non-partitioned and partitioned. Non-partitioned tables, have only one top-level segment
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups and partitions.
+
+Unique table identifier in Metastore Tables is combination of storage plugin, workspace and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies maximum level depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query.
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not active for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire tables (all files.)
+When you next issue the same command, Drill will scan only those files added since the previous run.
+The command will return the message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups which
+do not contain data which your query needs. If you add new files or directories, and do not rerun `ANALYZE TABLE`,
+then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so that
+ Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the alpha phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and unpack
+ archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+ mkdir /tmp/lineitem
+ mkdir /tmp/lineitem/s1
+ mkdir /tmp/lineitem/s2
+ cp TPCH/lineitem /tmp/lineitem/s1
+ cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+| EXPR$0 |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains group scan with `usedMetastore = false`:
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02 DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+| ok | summary |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, its metadata is collected and stored, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02 DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name |Type |Nullable |Description |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE` |VARCHAR |YES |Table data type: `PARQUET`, `CSV`, `JSON` |
+|`LOCATION` |VARCHAR |YES |Table location: `/tmp/nation` |
+|`NUM_ROWS` |BIGINT |YES |The total number of rows in all files of the table. Null if not known |
+|`LAST_MODIFIED_TIME` |TIMESTAMP |YES |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run. |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE | LOCATION | NUM_ROWS | LAST_MODIFIED_TIME |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL | dfs.tmp | lineitem | TABLE | PARQUET | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name |Type |Nullable |Description |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT` |VARCHAR |YES |Column default value. |
+|`COLUMN_FORMAT` |VARCHAR |YES |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion). |
+|`NUM_NULLS` |BIGINT |YES |The number of rows which contain nulls for this column. |
+|`MIN_VAL` |VARCHAR |YES |The minimum value of the column expressed as a string. For example: `'-273'`. |
+|`MAX_VAL` |VARCHAR |YES |The maximum value of the column expressed as a string. For example: `'100500'`. |
+|`NDV` |FLOAT8 |YES |Number of distinct values in column, expressed in Double. |
+|`EST_NUM_NON_NULLS` |FLOAT8 |YES |Estimated number of non null values, expressed in Double. |
+|`IS_NESTED` |BIT |NO |If column is nested. Nested columns are extracted from columns with struct type. |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL | NDV | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL | dfs.tmp | lineitem | dir0 | 1 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | s1 | s2 | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_orderkey | 2 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 6000000 | 1499876.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_partkey | 3 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 200000 | 199857.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_suppkey | 4 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 10000 | 10001.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_linenumber | 5 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 7 | 7.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_quantity | 6 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | 0 | 1.0 | 50.0 | 50.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_extendedprice | 7 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | 0 | 901.0 | 104949.5 | 933142.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_discount | 8 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | 0 | 0.0 | 0.1 | 11.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_tax | 9 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | 0 | 0.0 | 0.08 | 9.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_returnflag | 10 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | A | R | 3.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_linestatus | 11 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | F | O | 2.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_shipdate | 12 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | 0 | 694310400000 | 912470400000 | 2526.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_commitdate | 13 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | 0 | 696816000000 | 909792000000 | 2466.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_receiptdate | 14 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | 0 | 694483200000 | 915062400000 | 2554.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_shipinstruct | 15 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | COLLECT COD | TAKE BACK RETURN | 4.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_shipmode | 16 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | AIR | TRUCK | 7.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_comment | 17 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | Tiresias | zzle? slyly final platelets sleep quickly. | 4586320.0 | 1.200243E7 | false |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name |Type |Nullable |Description |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG` |VARCHAR |YES |Table catalog (currently we have only one catalog): `DRILL`. |
+|`TABLE_SCHEMA` |VARCHAR |YES |Table schema: `dfs.tmp`. |
+|`TABLE_NAME` |VARCHAR |YES |Table name: `nation`. |
+|`METADATA_KEY` |VARCHAR |YES |Top level segment key, the same for all nested segments and partitions: `part_int=3`. |
+|`METADATA_TYPE` |VARCHAR |YES |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition. |
+|`METADATA_IDENTIFIER` |VARCHAR |YES |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table. |
+|`PARTITION_COLUMN` |VARCHAR |YES |Partition column name: `part_varchar`. |
+|`PARTITION_VALUE` |VARCHAR |YES |Partition column value: `g`. |
+|`LOCATION` |VARCHAR |YES |Segment location, `null` for partitions: `/tmp/nation/part_int=3`. |
+|`LAST_MODIFIED_TIME` |TIMESTAMP |YES |Last modification time. |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE | LOCATION | LAST_MODIFIED_TIME |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL | dfs.tmp | lineitem | s2 | SEGMENT | s2 | `dir0` | s2 | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL | dfs.tmp | lineitem | s1 | SEGMENT | s1 | `dir0` | s1 | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
+
+Table metadata may be dropped using `ANALYZE TABLE DROP METADATA` command:
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
++------+----------------------------------------+
+| ok | summary |
++------+----------------------------------------+
+| true | Metadata for table [lineitem] dropped. |
++------+----------------------------------------+
+1 row selected (0.291 seconds)
+```
+
+### Collecting metadata for specific set of columns
+
+For the case when metadata for several columns should be computed and stored into the Metastore, the following command may be used:
+
+```
+apache drill (information_schema)> ANALYZE TABLE dfs.tmp.lineitem COLUMNS(l_orderkey, l_partkey) REFRESH METADATA;
++------+-------------------------------------------------------------+
+| ok | summary |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (94.1 seconds)
+```
+
+Now, check, that metadata is collected only for specified columns (`MIN_VAL`, `MAX_VAL`, `NDV`, etc.), but all
+ columns are present:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL | NDV | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| DRILL | dfs.tmp | lineitem | dir0 | 1 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | s1 | s2 | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_orderkey | 2 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 6000000 | 1499876.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_partkey | 3 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 200000 | 199857.0 | 1.200243E7 | false |
+| DRILL | dfs.tmp | lineitem | l_suppkey | 4 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_linenumber | 5 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_quantity | 6 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_extendedprice | 7 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_discount | 8 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_tax | 9 | null | YES | DOUBLE | null | null | 0 | 2 | 0 | null | null | null | 24 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_returnflag | 10 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_linestatus | 11 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_shipdate | 12 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_commitdate | 13 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_receiptdate | 14 | null | YES | DATE | null | null | null | null | null | 10 | null | null | 10 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_shipinstruct | 15 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_shipmode | 16 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
+| DRILL | dfs.tmp | lineitem | l_comment | 17 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+17 rows selected (0.183 seconds)
+```
diff --git a/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md b/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
new file mode 100644
index 0000000..1294286
--- /dev/null
+++ b/_docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
@@ -0,0 +1,63 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date:
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
+ details on how to configure Iceberg Metastore implementation and its option descriptions, please refer to
+ [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
+
+### Iceberg Tables Location
+
+Iceberg tables will reside on the file system in the location based on
+Iceberg Metastore base location `drill.metastore.iceberg.location.base_path` and component specific location.
+If Iceberg Metastore base location is `/drill/metastore/iceberg`
+and tables component location is `tables`. Iceberg table for tables component
+will be located in `/drill/metastore/iceberg/tables` folder.
+
+Metastore metadata will be stored inside Iceberg table location provided
+in the configuration file. Drill table metadata location will be constructed
+based on specific component storage keys. For example, for `tables` component,
+storage keys are storage plugin, workspace and table name: unique table identifier in Drill.
+
+Assume Iceberg table location is `/drill/metastore/iceberg/tables`, metadata for the table
+`dfs.tmp.nation` will be stored in the `/drill/metastore/iceberg/tables/dfs/tmp/nation` folder.
+
+Example of base Metastore configuration file `drill-metastore-override.conf`, where Iceberg tables will be stored in
+ hdfs:
+
+```
+drill.metastore.iceberg: {
+ config.properties: {
+ fs.defaultFS: "hdfs:///"
+ }
+
+ location: {
+ base_path: "/drill/metastore",
+ relative_path: "iceberg"
+ }
+}
+```
+
+### Metadata Storage Format
+
+Iceberg tables support data storage in three formats: Parquet, Avro, ORC. Drill metadata will be stored in Parquet files.
+This format was chosen over others since it is column oriented and efficient in terms of disk I/O when specific
+columns need to be queried.
+
+Each Parquet file will hold information for one partition. Partition keys will depend on Metastore
+component characteristics. For example, for tables component, partitions keys are storage plugin, workspace,
+table name and metadata key.
+
+Parquet files name will be based on UUID to ensure uniqueness. If somehow collision occurs, modify operation
+in Metastore will fail.
+
+### Iceberg metadata expiration
+
+Iceberg table generates metadata for each modification operation:
+snapshot, manifest file, table metadata file. Also when performing delete operation,
+previously stored data files are not deleted. These files with the time can occupy lots of space.
+Two table properties `write.metadata.delete-after-commit.enabled` and `write.metadata.previous-versions-max`
+control expiration process. Metadata files will be expired automatically if
+`write.metadata.delete-after-commit.enabled` is enabled.
diff --git a/_docs/query-data/060-querying-the-information-schema.md b/_docs/query-data/060-querying-the-information-schema.md
index d4e07fe..4e22ba3 100644
--- a/_docs/query-data/060-querying-the-information-schema.md
+++ b/_docs/query-data/060-querying-the-information-schema.md
@@ -1,6 +1,6 @@
---
title: "Querying the INFORMATION SCHEMA"
-date: 2019-04-05
+date: 2020-01-30
parent: "Query Data"
---
@@ -17,6 +17,7 @@ context. You can query the following INFORMATION_SCHEMA tables:
* COLUMNS
* VIEWS
* FILES
+ * PARTITIONS
## SCHEMATA
@@ -73,7 +74,11 @@ type) for each column in each table or view.
The VIEWS table returns the name and definition for each view in your
databases. Note that file schemas are the canonical repository for views in
Drill. Depending on how you create a view, the may only be displayed in Drill
-after it has been used.
+after it has been used.
+
+## PARTITIONS
+
+The PARTITIONS table returns information about table partitions including partition columns, partition keys, etc. for each table.
## FILES
diff --git a/_docs/sql-reference/sql-commands/005-supported-sql-commands.md b/_docs/sql-reference/sql-commands/005-supported-sql-commands.md
index 9d242e9..c7f5da2 100644
--- a/_docs/sql-reference/sql-commands/005-supported-sql-commands.md
+++ b/_docs/sql-reference/sql-commands/005-supported-sql-commands.md
@@ -1,29 +1,31 @@
---
title: "Supported SQL Commands"
-date: 2018-02-09 00:16:05 UTC
+date: 2020-01-28
parent: "SQL Commands"
---
The following table provides a list of the SQL commands that Drill supports,
with their descriptions and example syntax:
-| Command | Description | Syntax |
-|--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
-| [ALTER SESSION SET]({{site.baseurl}}/docs/set) | The SET command replaces the ALTER SESSION SET command in Drill version 1.3 and later. ALTER SESSION is now just an alias for the SET command. See [SET]({{site.baseurl}}/docs/set). | [ALTER SESSION] SET \`option_name` = value; |
-| [ALTER SYSTEM SET]({{site.baseurl}}/docs/alter-system) | Changes a system setting. The new setting persists across all sessions. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM SET \`option_name` = value; |
-| [ALTER SYSTEM RESET]({{site.baseurl}}/docs/alter-system) | Changes a system setting back to its default system setting. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM RESET \`option_name`; |
-| [ALTER SYSTEM RESET ALL]({{site.baseurl}}/docs/alter-system) | Changes all system settings back to their default system values. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM RESET ALL; |
-| [CREATE TABLE AS(CTAS)]({{site.baseurl}}/docs/create-table-as-ctas/) | Creates a new table and populates the new table with rows returned from a SELECT query. Use the CREATE TABLE AS (CTAS) statement in place of INSERT INTO. | CREATE TABLE name AS query; |
-| [CREATE TEMPORARY TABLE AS(CTTAS)]({{site.baseurl}}/docs/create-temp-table-as-cttas/) | Stores the results of a query in a temporary table. | CREATE TEMPORARY TABLE name AS query; |
-| [CREATE VIEW]({{site.baseurl}}/docs/create-view) | Creates a virtual structure for the result set of a stored query.- | CREATE [OR REPLACE] VIEW [workspace.]view_name [ (column_name [, ...]) ] AS query; |
-| [DESCRIBE ]({{site.baseurl}}/docs/describe) | Returns information about columns in a table or view. | DESCRIBE [workspace.]table_name |
-| [DROP VIEW]({{site.baseurl}}/docs/drop-view) | Removes a view. | DROP VIEW [workspace.]view_name ; |
-| [EXPLAIN PLAN FOR]({{site.baseurl}}/docs/explain) | Returns the physical plan for a particular query. | EXPLAIN PLAN FOR query; |
-| [EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR]({{site.baseurl}}/docs/explain) | Returns the logical plan for a particular query. | EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR query; |
-| [RESET]({{site.baseurl}}/docs/reset) | Changes a session setting back to its default system setting. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | [ALTER SESSION] RESET \`option_name`; |
-| [SELECT]({{site.baseurl}}/docs/select) | Retrieves data from tables and files. | [WITH subquery]SELECT column_list FROM table_name[WHERE clause] [GROUP BY clause][HAVING clause][ORDER BY clause]; |
-| [SET]({{site.baseurl}}/docs/set) | Changes a system setting for the duration of a session. A session ends when you quit the Drill shell. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | [ALTER SESSION] SET \`option_name` = value; |
-| [SHOW DATABASES]({{site.baseurl}}/docs/show-databases-and-show-schemas) | Returns a list of available schemas. Equivalent to SHOW SCHEMAS. | SHOW DATABASES; |
-| [SHOW FILES]({{site.baseurl}}/docs/show-files) | Returns a list of files in a file system schema. | SHOW FILES IN|FROM filesystem.\`schema_name`; |
-| [SHOW SCHEMAS]({{site.baseurl}}/docs/show-databases-and-show-schemas) | Returns a list of available schemas. Equivalent to SHOW DATABASES. | SHOW SCHEMAS; |
-| [SHOW TABLES]({{site.baseurl}}/docs/show-tables) | Returns a list of tables and views. | SHOW TABLES; |
-| [USE]({{site.baseurl}}/docs/use) | Change to a particular schema. When you opt to use a particular schema, Drill issues queries on that schema only. | USE schema_name; | [...]
+| Command | Description | Syntax |
+|-----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
+| [ALTER SESSION SET]({{site.baseurl}}/docs/set) | The SET command replaces the ALTER SESSION SET command in Drill version 1.3 and later. ALTER SESSION is now just an alias for the SET command. See [SET]({{site.baseurl}}/docs/set). | [ALTER SESSION] SET \`option_name` = value; |
+| [ANALYZE TABLE REFRESH METADATA]({{site.baseurl}}/docs/analyze-table-refresh-metadata) | Computes and stores the table schema and table statistics to the Drill Metastore. | ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}] REFRESH METADATA ['level' LEVEL] [{COMPUTE | ESTIMATE} | STATISTICS [...]
+| [ANALYZE TABLE COMPUTE STATISTICS]({{site.baseurl}}/docs/analyze-table-compute-statistics) | Computes statistics on Parquet data stored in tables and writes it to a JSON file in the `.stats.drill` directory. | ANALYZE TABLE [workspace.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT] |
+| [ALTER SYSTEM SET]({{site.baseurl}}/docs/alter-system) | Changes a system setting. The new setting persists across all sessions. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM SET \`option_name` = value; |
+| [ALTER SYSTEM RESET]({{site.baseurl}}/docs/alter-system) | Changes a system setting back to its default system setting. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM RESET \`option_name`; |
+| [ALTER SYSTEM RESET ALL]({{site.baseurl}}/docs/alter-system) | Changes all system settings back to their default system values. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | ALTER SYSTEM RESET ALL; |
+| [CREATE TABLE AS(CTAS)]({{site.baseurl}}/docs/create-table-as-ctas/) | Creates a new table and populates the new table with rows returned from a SELECT query. Use the CREATE TABLE AS (CTAS) statement in place of INSERT INTO. | CREATE TABLE name AS query; |
+| [CREATE TEMPORARY TABLE AS(CTTAS)]({{site.baseurl}}/docs/create-temp-table-as-cttas/) | Stores the results of a query in a temporary table. | CREATE TEMPORARY TABLE name AS query; |
+| [CREATE VIEW]({{site.baseurl}}/docs/create-view) | Creates a virtual structure for the result set of a stored query.- | CREATE [OR REPLACE] VIEW [workspace.]view_name [ (column_name [, ...]) ] AS query; |
+| [DESCRIBE ]({{site.baseurl}}/docs/describe) | Returns information about columns in a table or view. | DESCRIBE [workspace.]table_name |
+| [DROP VIEW]({{site.baseurl}}/docs/drop-view) | Removes a view. | DROP VIEW [workspace.]view_name ; |
+| [EXPLAIN PLAN FOR]({{site.baseurl}}/docs/explain) | Returns the physical plan for a particular query. | EXPLAIN PLAN FOR query; |
+| [EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR]({{site.baseurl}}/docs/explain) | Returns the logical plan for a particular query. | EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR query; |
+| [RESET]({{site.baseurl}}/docs/reset) | Changes a session setting back to its default system setting. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | [ALTER SESSION] RESET \`option_name`; |
+| [SELECT]({{site.baseurl}}/docs/select) | Retrieves data from tables and files. | [WITH subquery]SELECT column_list FROM table_name[WHERE clause] [GROUP BY clause][HAVING clause][ORDER BY clause]; |
+| [SET]({{site.baseurl}}/docs/set) | Changes a system setting for the duration of a session. A session ends when you quit the Drill shell. For a list of Drill options and their descriptions, see [Planning and Execution Options]{{site.baseurl/docs/planning-and-execution-options/}}. | [ALTER SESSION] SET \`option_name` = value; |
+| [SHOW DATABASES]({{site.baseurl}}/docs/show-databases-and-show-schemas) | Returns a list of available schemas. Equivalent to SHOW SCHEMAS. | SHOW DATABASES; |
+| [SHOW FILES]({{site.baseurl}}/docs/show-files) | Returns a list of files in a file system schema. | SHOW FILES IN|FROM filesystem.\`schema_name`; |
+| [SHOW SCHEMAS]({{site.baseurl}}/docs/show-databases-and-show-schemas) | Returns a list of available schemas. Equivalent to SHOW DATABASES. | SHOW SCHEMAS; |
+| [SHOW TABLES]({{site.baseurl}}/docs/show-tables) | Returns a list of tables and views. | SHOW TABLES; |
+| [USE]({{site.baseurl}}/docs/use) | Change to a particular schema. When you opt to use a particular schema, Drill issues queries on that schema only. | USE schema_name; | [...]
diff --git a/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md b/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
index 21fa711..50ad00e 100644
--- a/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
+++ b/_docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
@@ -1,158 +1,112 @@
---
title: "ANALYZE TABLE REFRESH METADATA"
parent: "SQL Commands"
-date: 2020-01-13
+date: 2020-01-30
---
Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
This metadata will be used when querying a table for more optimal plan creation.
+The Metastore is an Alpha feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
-To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
SET `metastore.enabled` = true;
+ ALTER SYSTEM SET `metastore.enabled` = true;
Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ and computes some metadata like MIN / MAX column values and NULLS COUNT designated as "metadata" to be able to
+ produce more optimizations like filter push-down, etc. If `planner.statistics.use` option is enabled, this command
+ will also calculate and store table statistics into Drill Metastore.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ in the same way as it is done during regular select.
+
## Syntax
The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
REFRESH METADATA ['level' LEVEL]
- [{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+ [{COMPUTE | ESTIMATE} | STATISTICS
[ SAMPLE number PERCENT ]]
## Parameters
*table_name*
-The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
- is temporary, the command fails and metadata is not collected and stored.
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, the table
+ is temporary or if you do not have permission to read the table, the command fails and metadata is not collected and stored.
*COLUMNS (col1, col2, ...)*
-Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+Optional names of the column(s) for which Drill will compute and store statistics. The stored schema will include all
+ table columns.
*COLUMNS NONE*
-Specifies to ignore collecting and storing metadata for all table columns.
+Drill will infer schema for all columns, but gather statistics for none of the columns.
*level*
-Optional varchar literal which specifies maximum level depth for collecting metadata.
-Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+Optional VARCHAR literal which specifies maximum level depth for collecting metadata.
+Possible values:
+
+- `TABLE` - metadata will be collected at table level (MIN / MAX column values within whole the table, etc.);
+- `SEGMENT` - metadata will be collected for every segment within the table (MIN / MAX column values within a specific
+ segment) + metadata at table level. Segment term here is used to abstract data part which may correspond to a
+ specific directory on the file system or partition in Hive table;
+- `PARTITION` - metadata will be collected for every partition within the table (MIN / MAX column values within a
+ specific partition) + file, segment and table metadata. Partition term is used here to abstract part of the data
+ where some column(s) have the same values (corresponds to existing Drill partitions for Parquet table). Is not
+ implemented in Drill 1.17;
+- `FILE` - metadata will be collected for every file within the table (MIN / MAX column values within a specific file) +
+ partition, segment metadata and table metadata;
+- `ROW_GROUP` - metadata will be collected for every row group within the table (MIN / MAX column values within a
+ specific row group) + file, partition, segment metadata and table metadata. Supported for Parquet tables only;
+- `ALL` - metadata will be collected for every splittable table part - row groups for parquet, files for regular file
+ storage formats, etc.
+Default is `ALL`.
*COMPUTE*
-Generates statistics for the table to be stored into the Metastore.
-If statistics usage is disabled (`planner.enable_statistics` is set to `false`), an error will be thrown when this clause is specified.
+Computes statistics for the table to be stored into the Metastore.
+If statistics usage is disabled (`planner.statistics.use` is set to `false`), an error will be thrown when this clause is specified.
*ESTIMATE*
-Generates estimated statistics for the table to be stored into the Metastore. Currently is not supported.
+Computes estimated statistics for the table to be stored into the Metastore. Currently is not supported.
*(column1, column2, ...)*
-The name of the column(s) for which Drill will generate statistics.
+The name of the column(s) for which Drill will compute statistics.
*SAMPLE*
Optional. Indicates that compute statistics should run on a subset of the data.
*number PERCENT*
-An integer that specifies the percentage of data on which to compute statistics. For example, if a table has 100 rows, `SAMPLE 50 PERCENT` indicates that statistics should be computed on 50 rows. The optimizer selects the rows at random.
-
-## Related Options
-
-- **metastore.enabled**
-Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
- to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
-- **metastore.metadata.store.depth_level**
-Specifies maximum level depth for collecting metadata. Default is `'ALL'`.
-- **metastore.retrieval.retry_attempts**
-Specifies the number of attempts for retrying query planning after detecting that query metadata is changed.
-If the number of retries was exceeded, query will be planned without metadata information from the Metastore. Default is 5.
-- **metastore.metadata.fallback_to_file_metadata**
-Allows using file metadata cache for the case when required metadata is absent in the Metastore. Default is true.
-- **metastore.metadata.use_schema**
-Enables schema usage, stored to the Metastore. Default is `true`.
-- **metastore.metadata.use_statistics**
-Enables statistics usage, stored in the Metastore, at the planning stage. Default is `true`.
-- **metastore.metadata.ctas.auto-collect**
-Specifies whether schema and / or column statistics will be automatically collected for every table after CTAS and CTTAS.
-This option is not active for now. Default is `'NONE'`.
-- **drill.exec.storage.implicit.last_modified_time.column.label**
-Sets the implicit column name for the last modified time (`lmt`) column. For internal usage when producing Metastore analyze.
-- **drill.exec.storage.implicit.row_group_index.column.label**
-Sets the implicit column name for the row group index (`rgi`) column. For internal usage when producing Metastore analyze.
-- **drill.exec.storage.implicit.row_group_length.column.label**
-Sets the implicit column name for the row group length (`rgl`) column. For internal usage when producing Metastore analyze.
-- **drill.exec.storage.implicit.row_group_start.column.label**
-Sets the implicit column name for the row group start (`rgs`) column. For internal usage when producing Metastore analyze.
+An integer that specifies the percentage of data on which to compute statistics. For example, if a table has 100 rows,
+ `SAMPLE 50 PERCENT` indicates that statistics should be computed on 50 rows. The optimizer selects the rows at random.
## Related Commands
-To drop table metadata from the Metastore, the following command may be used:
+Use the following command to remove a table from the Metastore:
ANALYZE TABLE [table_name] DROP [METADATA|STATISTICS] [IF EXISTS]
-It will not throw an exception for absent table metadata if `IF EXISTS` clause was specified.
-
- ANALYZE TABLE [workspace.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT]
-
-See [ANALYZE TABLE COMPUTE STATISTICS]({{site.baseurl}}/docs/analyze-table-compute-statistics).
-
-## Usage Notes
-
-### General Information
-
-- Currently `ANALYZE TABLE REFRESH METADATA` statement can compute and store metadata only for Parquet tables within `dfs` storage plugins.
-- For the case when `ANALYZE TABLE REFRESH METADATA` command is executed for the first time, whole table metadata will be collected and stored into Metastore.
-If analyze was already executed for the table, and table data wasn't changed, all further analyze commands wouldn't trigger table analyzing and message that table metadata is up to date will be returned.
-
-### Incremental analyze
+The command will fail if the table does not exist in the Metastore. Include the `IF EXISTS` clause to ignore a missing table.
-For the case when some table data was updated, Drill will try to execute incremental analyze - calculate metadata only for updated data and reuse required metadata from the Metastore.
+ ANALYZE TABLE [plugin.schema.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT]
-Incremental analyze wouldn't be produced for the following cases:
-- list of interesting columns specified in analyze is not a subset of interesting columns from the previous analyze;
-- specified metadata level differs from the metadata level in previous analyze.
+See [ANALYZE TABLE COMPUTE STATISTICS]({{site.baseurl}}/docs/analyze-table-compute-statistics).
-### Metadata usage
+ REFRESH TABLE METADATA [ COLUMNS ( column1, column2...) | NONE ] table_path
-Drill provides the ability to use metadata obtained from the Metastore at the planning stage to prune segments, files
- and row groups.
+For the case when table metadata was stored into the Drill Metastore, Parquet table metadata cache files, wouldn't be
+ used for the same table during query execution if all required metadata is present and is not outdated.
-Tables metadata from the Metastore is exposed to `INFORMATION_SCHEMA` tables (if Metastore usage is enabled).
-
-The following tables are populated with table metadata from the Metastore:
-
-`TABLES` table has the following additional columns populated from the Metastore:
-
-- `TABLE_SOURCE` - table data type: `PARQUET`, `CSV`, `JSON`
-- `LOCATION` - table location: `/tmp/nation`
-- `NUM_ROWS` - number of rows in a table if known, `null` if not known
-- `LAST_MODIFIED_TIME` - table's last modification time
-
-`COLUMNS` table has the following additional columns populated from the Metastore:
-
-- `COLUMN_DEFAULT` - column default value
-- `COLUMN_FORMAT` - usually applicable for date time columns: `yyyy-MM-dd`
-- `NUM_NULLS` - number of nulls in column values
-- `MIN_VAL` - column min value in String representation: `aaa`
-- `MAX_VAL` - column max value in String representation: `zzz`
-- `NDV` - number of distinct values in column, expressed in Double
-- `EST_NUM_NON_NULLS` - estimated number of non null values, expressed in Double
-- `IS_NESTED` - if column is nested. Nested columns are extracted from columns with struct type.
-
-`PARTITIONS` table has the following additional columns populated from the Metastore:
-
-- `TABLE_CATALOG` - table catalog (currently we have only one catalog): `DRILL`
-- `TABLE_SCHEMA` - table schema: `dfs.tmp`
-- `TABLE_NAME` - table name: `nation`
-- `METADATA_KEY` - top level segment key, the same for all nested segments and partitions: `part_int=3`
-- `METADATA_TYPE` - `SEGMENT` or `PARTITION`
-- `METADATA_IDENTIFIER` - current metadata identifier: `part_int=3/part_varchar=g`
-- `PARTITION_COLUMN` - partition column name: `part_varchar`
-- `PARTITION_VALUE` - partition column value: `g`
-- `LOCATION` - segment location, `null` for partitions: `/tmp/nation/part_int=3`
-- `LAST_MODIFIED_TIME` - last modification time
-
-## Limitations
+## Usage Notes
-This feature is currently in the alpha phase (preview, experimental) for Drill 1.17 and only applies to Parquet
- tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+For more detailed information on Drill Metastore and its usage please refer to [Using Drill Metastore]({{site.baseurl}}/docs/using-drill-metastore).