You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/01/13 14:46:39 UTC

[GitHub] [drill] vvysotskyi opened a new pull request #1953: Add docs for Drill Metastore

vvysotskyi opened a new pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366696000

##########
File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
##########
@@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+ SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+ ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+ REFRESH METADATA ['level' LEVEL]
+ [{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+ [ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
+If statistics usage is disabled (`planner.enable_statistics` is set to `false`), an error will be thrown when this clause is specified.
+
+*ESTIMATE*
+Generates estimated statistics for the table to be stored into the Metastore. Currently is not supported.
+
+*(column1, column2, ...)*
+The name of the column(s) for which Drill will generate statistics.
+
+*SAMPLE*
+Optional. Indicates that compute statistics should run on a subset of the data.
+
+*number PERCENT*
+An integer that specifies the percentage of data on which to compute statistics. For example, if a table has 100 rows, `SAMPLE 50 PERCENT` indicates that statistics should be computed on 50 rows. The optimizer selects the rows at random.
+
+## Related Options
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies maximum level depth for collecting metadata. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+Specifies the number of attempts for retrying query planning after detecting that query metadata is changed.
+If the number of retries was exceeded, query will be planned without metadata information from the Metastore. Default is 5.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using file metadata cache for the case when required metadata is absent in the Metastore. Default is true.
+- **metastore.metadata.use_schema**
+Enables schema usage, stored to the Metastore. Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables statistics usage, stored in the Metastore, at the planning stage. Default is `true`.
+- **metastore.metadata.ctas.auto-collect**
+Specifies whether schema and / or column statistics will be automatically collected for every table after CTAS and CTTAS.
+This option is not active for now. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. For internal usage when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. For internal usage when producing Metastore analyze.
+
+## Related Commands
+
+To drop table metadata from the Metastore, the following command may be used:
+
+ ANALYZE TABLE [table_name] DROP [METADATA|STATISTICS] [IF EXISTS]
+
+It will not throw an exception for absent table metadata if `IF EXISTS` clause was specified.
+
+ ANALYZE TABLE [workspace.]table_name COMPUTE STATISTICS [(column1, column2,...)] [SAMPLE number PERCENT]
+
+See [ANALYZE TABLE COMPUTE STATISTICS]({{site.baseurl}}/docs/analyze-table-compute-statistics).
+
+## Usage Notes
+
+### General Information
+
+- Currently `ANALYZE TABLE REFRESH METADATA` statement can compute and store metadata only for Parquet tables within `dfs` storage plugins.
+- For the case when `ANALYZE TABLE REFRESH METADATA` command is executed for the first time, whole table metadata will be collected and stored into Metastore.
+If analyze was already executed for the table, and table data wasn't changed, all further analyze commands wouldn't trigger table analyzing and message that table metadata is up to date will be returned.
+
+### Incremental analyze
+
+For the case when some table data was updated, Drill will try to execute incremental analyze - calculate metadata only for updated data and reuse required metadata from the Metastore.
+
+Incremental analyze wouldn't be produced for the following cases:
+- list of interesting columns specified in analyze is not a subset of interesting columns from the previous analyze;
+- specified metadata level differs from the metadata level in previous analyze.
+
+### Metadata usage
+
+Drill provides the ability to use metadata obtained from the Metastore at the planning stage to prune segments, files
+ and row groups.
+
+Tables metadata from the Metastore is exposed to `INFORMATION_SCHEMA` tables (if Metastore usage is enabled).
+
+The following tables are populated with table metadata from the Metastore:
+
+`TABLES` table has the following additional columns populated from the Metastore:
+
+- `TABLE_SOURCE` - table data type: `PARQUET`, `CSV`, `JSON`
+- `LOCATION` - table location: `/tmp/nation`
+- `NUM_ROWS` - number of rows in a table if known, `null` if not known
+- `LAST_MODIFIED_TIME` - table's last modification time
+
+`COLUMNS` table has the following additional columns populated from the Metastore:
+
+- `COLUMN_DEFAULT` - column default value
+- `COLUMN_FORMAT` - usually applicable for date time columns: `yyyy-MM-dd`
+- `NUM_NULLS` - number of nulls in column values
+- `MIN_VAL` - column min value in String representation: `aaa`
+- `MAX_VAL` - column max value in String representation: `zzz`
+- `NDV` - number of distinct values in column, expressed in Double
+- `EST_NUM_NON_NULLS` - estimated number of non null values, expressed in Double
+- `IS_NESTED` - if column is nested. Nested columns are extracted from columns with struct type.
+
+`PARTITIONS` table has the following additional columns populated from the Metastore:
+
+- `TABLE_CATALOG` - table catalog (currently we have only one catalog): `DRILL`
+- `TABLE_SCHEMA` - table schema: `dfs.tmp`
+- `TABLE_NAME` - table name: `nation`
+- `METADATA_KEY` - top level segment key, the same for all nested segments and partitions: `part_int=3`

Review comment:
(This is the `PARTITON_KEY` or `PARTITIONS`.`KEY_VALUE`.

The format of the value is the same as for Hive. For this to work (see comment above), we somehow have to know that `part_int` is a field name, which we don't know unless the user tells us. Otherwise, this would be `dir0=2019` without knowing that this is really a year.

Or, have we somehow started using Hive-format partitions without telling the user how to specify the key name?

Suggestion: please review the Hive metastore schema; they have spent many years working out these issues. Lets reuse their knowledge, changing it where needed for Drill (or where Hive made mistakes. See [this diagram](https://community.cloudera.com/t5/Support-Questions/Is-there-a-updated-Hive-1-2-1-Metastore-ER-Diagram/td-p/157027).)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380210657

##########
File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
##########
@@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+ SET `metastore.enabled` = true;
+ ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query.
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+ mkdir /tmp/lineitem
+ mkdir /tmp/lineitem/s1
+ mkdir /tmp/lineitem/s2
+ cp TPCH/lineitem /tmp/lineitem/s1
+ cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+| EXPR$0 |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02 DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+| ok | summary |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02 DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:

Review comment:
This is a definitive list, moved it from the tutorial.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380276160

Review comment:
Thanks, updated the docs as proposed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373004866
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
 
 Review comment:
   `ALTER SESSION` may be omitted, by default it will be set at a session level.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374454291

##########
File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
##########
@@ -0,0 +1,69 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For Drill 1.17,
+ this is default Drill Metastore implementation. For details on how to configure Iceberg Metastore implementation and
+ its option descriptions, please refer to [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
+
+{% include startnote.html %}
+Iceberg table supports concurrent writes and transactions but they are only effective on file systems that support
+ atomic rename.
+If the file system does not support atomic rename, it could lead to inconsistencies during concurrent writes.
+{% include endnote.html %}
+
+### Iceberg Tables Location
+
+Iceberg tables will reside on the file system in the location based on
+Iceberg Metastore base location `drill.metastore.iceberg.location.base_path` and component specific location.
+If Iceberg Metastore base location is `/drill/metastore/iceberg`
+and tables component location is `tables`. Iceberg table for tables component
+will be located in `/drill/metastore/iceberg/tables` folder.
+
+Metastore metadata will be stored inside Iceberg table location provided
+in the configuration file. Drill table metadata location will be constructed
+based on specific component storage keys. For example, for `tables` component,
+storage keys are storage plugin, workspace and table name: unique table identifier in Drill.
+
+Assume Iceberg table location is `/drill/metastore/iceberg/tables`, metadata for the table
+`dfs.tmp.nation` will be stored in the `/drill/metastore/iceberg/tables/dfs/tmp/nation` folder.
+
+Example of base Metastore configuration file `drill-metastore-override.conf`, where Iceberg tables will be stored in
+ hdfs:
+
+```
+drill.metastore.iceberg: {
+ config.properties: {
+ fs.defaultFS: "hdfs:///"
+ }
+
+ location: {
+ base_path: "/drill/metastore",
+ relative_path: "iceberg"
+ }
+}
+```
+
+### Metadata Storage Format
+
+Iceberg tables support data storage in three formats: Parquet, Avro, ORC. Drill metadata will be stored in Parquet files.
+This format was chosen over others since it is column oriented and efficient in terms of disk I/O when specific
+columns need to be queried.
+
+Each Parquet file will hold information for one partition. Partition keys will depend on Metastore
+component characteristics. For example, for tables component, partitions keys are storage plugin, workspace,
+table name and metadata key.
+
+Parquet files name will be based on UUID to ensure uniqueness. If somehow collision occurs, modify operation
+in Metastore will fail.
+
+### Iceberg metadata expiration
+
+Iceberg table generates metadata for each modification operation:

Review comment:
What is "Iceberg table"? Is this an Iceberg concept? Drill information schema tables stored in Iceberg? Metadata for user tables?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374440065

Review comment:
Enables the Drill query planner to use table and column statistics stored in the Metastore.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374448673
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
+
+Table metadata may be dropped using `ANALYZE TABLE DROP METADATA` command:
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
++------+----------------------------------------+
+|  ok  |                summary                 |
++------+----------------------------------------+
+| true | Metadata for table [lineitem] dropped. |
++------+----------------------------------------+
+1 row selected (0.291 seconds)
+```
+
+### Collecting metadata for specific set of columns
 
 Review comment:
   Collecting --> Collect
   
   (Still in the tutorial?)
   
   If so, reword to say, "Next lets gather metadata for a subset of the columns in the lineitem table. You would do this when..."

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374438429

Review comment:
This is a bit to vague to be useful. First, describe how the Metastore describes a table. Maybe, "A table is assumed to be made op of one or more files which reside in one or more directories. Every table described by the Metastore must reside in a top-level folder; the Metastore cannot describe bare files. (True?)

If a table consists of a single directory, then it is non-partitioned. The single directory can contain any number of files. Larger tables tend to have subdirectories. Each subdirectory is a partition and such table are called "partitioned."

It looks like the material is trying to then say how the information is stored. I would describe that separately since there is nothing a user can do with the information.

Instead, it would be better to show an example. We do hae a way to query metastore data? Then say:

"The following shows metadata for a non-partitioned table:

```
SELECT ...
<show output>
```

Partitioned table use *partition keys*: the directory name describes data within that directory. For example, the "2019" folder holds data for the year 2019. The Metastore contains an entry for each partition. (Show an example.)

Partitions can be nested: we might have subdirectories 01-12 within the 2019 folder. (Explain how this appears when queried.)

The Metastore assumes your directory names follow a convention. You may be familiar with the Hive 'col=value` format. Drill instead uses (describe what it is.)"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380209529

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380159080

Review comment:
Thanks, added the comment you have proposed, now it looks really better.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380159969

Review comment:
Thanks, replaced it.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380278702

Review comment:
Thanks, removed this section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371682359
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
 
 Review comment:
   Thanks, replaced.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374430816

Review comment:
Unclear: "Drill can query a table whether that table
has a Metastore entry or not." Do we mean that Drill can query the Metastore to determine if table has an entry in the Metastore or not? If so, this seems sort of obvious.

Or do we mean "Metastore entries are optional. If you query a table without a Metastore entry, Drill works with that table just as if the Metastore was disabled." Reword as needed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374445060

Review comment:
Does this work for a distributed Drill? Or, only for a single-node Drill on, say, a laptop?

Make clearer: "This example assumes you are running a single-node Drill on your laptop. We will store the lineitem table data in /tmp as follows:"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374444417

Review comment:
Would be better to change this to a Tutorial and put it before the detailed nuts-and-bolts information.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379573331

Review comment:
It specified metadata kind. Reworded it.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380146051

Review comment:
Thanks, changed the title name.
It is a good idea to explain the process.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371864788

Review comment:
Thanks, updated as you have suggested.
I think having a single command is OK, the user may use it without digging into these details if it is not essential for user.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366693328

Review comment:
The number of distinct column values. (Is NULL considered a distinct value? Please define this, it makes a huge difference for low-NDV columns.

Perhaps reformat this as a table:

| Table Name | Column Name | Type | NULL? | Description |
| ---------------- | ------------------- | ------ | ------ | --------------- |
| TABLE | LOCATION | VARCHAR | NO | Location of the table... |
| COLUMNS | TABLE_NAME | VARCHAR | NO |Name of the table for this column. |
| | NDV | DOUBLE | YES | Number of distinct values... |

Then, since we are declaring nullability, it is clear that, say the table location is required, but the NDV can be NULL if not known.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366681625
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
 
 Review comment:
   varchar --> VARCHAR
   
   (Since this is a quoted string, can we make it user friendly? "ROW_GROUP" --> "ROW GROUP" (no underscore.) Actually, why can't this be a keyword?
   
   Please define the terms, especially `SEGMENT`. Also, what is the advantage of one level vs. another?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371899388

Review comment:
No, it is not consistent with HMS. Storage format sounds confusing for the case when metastore will work with other storage plugins. Name is taken from the handler which will be used to handle specific storage format.
`LOCATION` here is the actual table location, resolved using workspace. No, it is not the value specified in `ANALYZE TABLE`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371850639

Review comment:
Thanks, `plugin.schema` more correct. Workspaces resolution is the same as for regular select.
Regarding removing plugin or schema, yes, metadata will still be present in the schema.
I think the solution is to introduce a new command which will walk through metastore tables and remove metadata for non-existent tables.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373453620

Review comment:
```suggestion
The command will return the following message if table statistics are up-to-date:
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373455456

##########
File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
##########
@@ -0,0 +1,378 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-30
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is an Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+ SET `metastore.enabled` = true;
+ ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ and computes some metadata like MIN / MAX column values and NULLS COUNT designated as "metadata" to be able to
+ produce more optimizations like filter push-down, etc. If `planner.statistics.use` option is enabled, this command
+ will also calculate and store table statistics into Drill Metastore.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
+ in the same way as it is done during regular select.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like udfs, storage plugins, etc. be added in future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to `BaseTableMetadata` class.
+
+Table can be non-partitioned and partitioned. Non-partitioned tables, have only one top-level segment
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups and partitions.
+
+Unique table identifier in Metastore Tables is combination of storage plugin, workspace and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies maximum level depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query.
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not active for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire tables (all files.)
+When you next issue the same command, Drill will scan only those files added since the previous run.
+The command will return the message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups which
+do not contain data which your query needs. If you add new files or directories, and do not rerun `ANALYZE TABLE`,
+then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so that
+ Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the alpha phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and unpack
+ archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+ mkdir /tmp/lineitem
+ mkdir /tmp/lineitem/s1
+ mkdir /tmp/lineitem/s2
+ cp TPCH/lineitem /tmp/lineitem/s1
+ cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+| EXPR$0 |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains group scan with `usedMetastore = false`:
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02 DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+| ok | summary |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, its metadata is collected and stored, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02 DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+| ok | summary |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.

Review comment:
Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it disabled, info tables won't contain Metastore metadata.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373456108
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,63 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date:
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
 
 Review comment:
   Please add:
   For Drill 1.17, this is default Drill Metastore implementation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371842074

Review comment:
Removed part "For internal usage". These options may be useful for the case when the existing table column matches its default value.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371151569
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
 
 Review comment:
   Metastore usage may be enabled either at a session or system level. Updated docs to reflect it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366685757

Review comment:
(How does this option relate to the planner option above? Must I enable both? Why would I disable one or the other?

I wonder, should this stuff be in an appendix somewhere and not here? As a (somewhat informed) reader, I find myself becoming more and more confused about how this stuff even works! Maybe only list the key things here: options that represent knobs that the user might actually want to twiddle.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366679238
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
 
 Review comment:
   (Here it is not clear if the command gathers schema and statistics? Or, only schema by default?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373017272
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md
 ##########
 @@ -99,7 +99,7 @@ Controls the 'compression' factor for the TDigest algorithm used for histogram s
 
 ## Reserved Keywords
 
-The ANALYZE TABLE statement introduces the following reserved keywords:  
+The ANALYZE TABLE COMPUTE STATISTICS statement introduces the following reserved keywords:  
 
 Review comment:
   Command wasn't changed, just added additional words which already are part of the command to distinguish it from another command which starts similarly.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366688826

Review comment:
If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update statistics, called "incremental analysis." Incremental analysis will compute metadata only for files and partitions changed since the last analysis.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371931973

Review comment:
This is common for partition and segment. There are additional fields, specific for partition itself, named as `PARTITION_COLUMN` and `PARTITION_VALUE`.

Initially, we have looked into the hive structure, but Drill representation differs from that one and should be suitable not only for Hive tables but for other storages. For example, we may store metadata for every level: partition, file, row group, etc.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371789711

Review comment:
Added a reference to file metadata cache page.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374639905

Review comment:
Yes, we are using storage plugin config name. `dfs` is used as config name in many examples in Drill docs but yes, we can rename to `my-dfs` if it would be clearer.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366686704

Review comment:
Use the following command to remove a table from the Metastore:

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380192497

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374777766

Review comment:
Thanks, reworded.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379541595

Review comment:
Thanks, separated these two concepts and added links to iceberg documentation.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371909842

Review comment:
This is an estimated nulls number, it is the value used when computing column statistics (not metadata, where nulls_number is precise).

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380275860

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373451748

Review comment:
What does it mean `non active`? Not implemented?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380199004

Review comment:
Thanks, added sentence with the reference to corresponding instruction.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374434708

Review comment:
The term "Metastore Tables" is misleading. I read it like "Hive Tables": the relational DB tables which make up Hives internal DB schema. I think you mean, "Table Metadata": the metadata about tables which is stored in the Metastore.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374440213

Review comment:
If not implemented, don't document it. Save the documentation for when it is implemented. (That is, we don't document future features.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374430161
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
 
 Review comment:
   Suggestion: insert an introduction section before the Enabling section. Else, it is not clear what the Metastore give me and why I'd want to enable it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373538164

Review comment:
Thanks, added missing word.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373454802

Review comment:
```suggestion
Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374439568

Review comment:
Maximum depth of what? Partition directory structure? Schema JSON-like MAP structure?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373537401

Review comment:
Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374441648

Review comment:
Default: `lmt`. Here and below.

Note that this is a hack that should be fixed. Filed a JIRA with this and other issues.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371858041

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373014394

Review comment:
Updated comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374450331

Review comment:
This worries me. S3 does not support atomic renames: renames make a copy. Will metastore work on S3? If not, we should call out that Drill 1.17 supports only the local file system, HDFS and similar systems. Drill 1.17 does not support S3 because S3 does not provide atomic rename.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366693983

Review comment:
(The data model here does not seem to work. Nested inside of what? I would need a foreign key into this table: `(COLUMNS.TABLE_NAME, COLUMNS.PARENT_COLUMN_NAME) --> (COLUMNS.TABLE_NAME, COLUMNS.NAME) for child/parent relationships.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366685134

Review comment:
(Can we eliminate the `metadata` here? The Metastore only works with metadata, so no need to add an extra layer of names.

So far in this document, we've said nothing about a "file metadata cache" until now. We should describe it before we explain options that reference it.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r372428487

Review comment:
Reformated as a table.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366691711

Review comment:
(Is the above new? If not, can we point to the existing documentation?

Is the above consistent with the Hive Metastore? If so, we should note that these tables are "a subset of the Hive schema".)

(`TABLE_SOURCE` is a confusing name, sounds the same as `LOCATION`. `STORED_AS` or `FORMAT` would be better.)

`TABLE_SOURCE` - The table storage format. (From where? Storage plugin name? Are CSV, TSV, CSVH all treated as CSV? If I name my format plugin "csv", will the source here be "csv" or will Drill somehow automagically figure out it is "CSV"?) In Drill 1.17, only `PARQUET` is supported.)
`LOCATION` - Table location in the form `plugin.schema./directory/path`. (Is this the same as the value specified in `ANALYZE TABLE` or, will we doctor it up somehow, such as to resolve workspace or `USE` references?)
`NUM_ROW` - the total number of rows in all files of the table. Null if not known.
`LAST_MODIFIED_TIME` - timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366690334

Review comment:
Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups, Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups which do not contain data which your query needs. If you add new files or directories, and do not rerun `ANALYZE TABLE`, then Drill will assume that the new data might contain the data you want and will scan them. Periodically rerun `ANALYZE TABLE` so that Drill can skip new files and tables when possible.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366684283

Review comment:
Enables the Drill Metastore and the `ANALYZE TABLE` command.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379568993

Review comment:
Thanks, replaced. Currently, user can delete only metadata for an existing table. Added this info also.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371752852

Review comment:
Thanks, renamed section topic. Regarding the scope of options, for some cases, it is useful to disable metastore at session level or specify set other options.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371904606

Review comment:
It will be possible to set for a user with a schema provisioning feature when running analyze if https://issues.apache.org/jira/browse/DRILL-7477 is implemented.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366692613

Review comment:
(Please explain the format, or point to "provided schema" documentation for it.")

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374438898

Review comment:
Again, turn this around in terms of what the user does.

"A traditional database divides tables into schemas and tables. Drill can connect to any number of data sources, each of which may have its own schema. As a result, the Metastore labels tables with a combination of (plugin configuration name, workspace name, table name). Note that if you rename any of these items, you must delete and recreate the table's Metadata entry. (We can do that, can't we?)"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] bbevens commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

bbevens commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-582080790
 
 
   Tested HTML build locally. Merging and I'll build and publish the site.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374443038

Review comment:
Move to the previous section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373460855
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,63 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date:
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
+ details on how to configure Iceberg Metastore implementation and its option descriptions, please refer to
+ [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
 
 Review comment:
   Please include information about Iceberg Metastore limitations:
   ```
   Note: Iceberg table supports concurrent writes and transactions 
   but they are only effective on file systems that support atomic rename.
   If file system does not support atomic rename, it could lead to inconsistencies during concurrent writes. 
   
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373450777

Review comment:
```suggestion
Specifies maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
```
There are now level options above...

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374641533

Review comment:
User can create it beforehand but Drill will create it himself if it is absent, the same as done for Dynamic Metastore. If user creates it himself, user that running Drillbit should have all the permissions.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374778009

Review comment:
Thanks, done.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371922729

Review comment:
As I mentioned above, these fields also are present in `TABLES` table.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371835861

Review comment:
Thanks, updated text. Yes, that's correct that it will be supported only for regular tables. This and /or meant schema without statistics. With enumerated possible values, it should be clear.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380229001

Review comment:
Good idea, I have extracted definitive info and moved it to the separate section. Examples are placed under tutorial section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374640712

Review comment:
The same as above, dfs is storage config name, user can give any name, so I hope we don't have a problem. If user creates fs storage config with name `my-fs`, storage path will be the following: `my-fs/tmp/lineitem`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-580811056
 
 
   @vvysotskyi thanks for writing such good documentation.
   +1

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379551973

Review comment:
Thanks, replaced as you proposed, but also left mentioning that we have metadata about segments, files, row groups, partitions since it wasn't described in this doc yet.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379576801

Review comment:
Thanks, specified default values.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379576091

Review comment:
Thanks, removed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371754837

Review comment:
I think the existing comment more informative.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374453026

Review comment:
Should the user create the /drill/metastore directory ahead of time? With which permissions? Which user does Drill use to access HDFS? If HDFS is secure, where to I put my Kerberos ticket?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373457265

Review comment:
You can add example, how it looks like and note, that user can change the implementation.
You can note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom implementation can be added by placing jar into classpath which has implementation of `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366695302

Review comment:
(Please explain what a SEGMENT might be and how it differs from a partition.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374436719

Review comment:
This is user documentation; users probably won't look at source code to figure out what's what. A better approach is to say, "The detailed metadata schema is described (here). You can try out the metadata, and get a sense of what is available, by using the Metadata tutorial." (We do have one, complete with a sample data set? If not, we should add one.)

The class-level information could go into a "Metastore API Guide" targeted at developers.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371922057

Review comment:
This table contains all partitions for all tables. Also, this set of columns is common for almost all tables from `information_schema`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380230120
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
 
 Review comment:
   Thanks, done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373567421
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,63 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date:
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
+ details on how to configure Iceberg Metastore implementation and its option descriptions, please refer to
+ [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
 
 Review comment:
   Thanks, added.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373016273

Review comment:
Added a more detailed comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366692861

Review comment:
The minimum value of the column expressed as a string. For example: `'10'`.

(And below, replace "max" with "maximum using the above format.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373534723

Review comment:
Yes, is not implemented. Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374447766
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
 
 Review comment:
   Move the definition to the earlier section per an earlier comment. Here, say "The sample lineitem table has two partitions. The PARTITIONS table..."

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374446251

Review comment:
Remove the comma: "Once table metadata..." Better to be less passive.

"Now that we've collected table metadata, we can use it when we query the table. by checking the ... entry:"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374445538

Review comment:
This is tutorial, not a general description. So, Computing and storing --> Compute and store.

Better "Compute table metadata and store in the Drill Metastore"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373029314

Review comment:
This PR only adds docs to describe how Drill Metastore works, if something in the code should be changed or updated, let's discuss it separately.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373454640

Review comment:
Note, that you first need to enable Metastore.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366682430
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
 
 Review comment:
   Generates --> Computes
   
   (The command is `COMPUTE` so we should use that word here instead of "generate.")

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374641961

Review comment:
Yes, it's an Iceberg concept.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380250105

Review comment:
Good point! Added sentence before this one about configuration files and added specified that the above is the configuration property.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373565916

Review comment:
Thanks, added.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373533767

Review comment:
Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374433599

Review comment:
These are two large concepts, mentioned in passing. Better to deal with each of them.

Metastore Implementations

The Metastore offers an API which allows for any number of implementations. See (what?) for a description of the API.

The default implementation is the Iceberg Metastore which provides (list the feeatures and how to use it. What does "Iceberg" mean? Where does it reside? And so on.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374446627

Review comment:
What is the goal of this section? Still part of the tutorial? Then the user does not "Expose", the user "queries.". So a better title might be "Inspect the Metastore using INFORMATION_SCHEMA tables"

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374451248

Review comment:
"table for tables component" very confusing. The user is missing some key information. Maybe:

Within the metastore directory, the Metastore stores the data for each table in a separate Iceberg file. All table metadata resides in a single file. Table metadata resides in the iceberg/tables directory.

Let's write this assuming that the user changed the root folder. So, maybe: `${drill.metastore.iceberg.location.base_path}/tables`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366678258
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
 
 Review comment:
   (Ah, if it is an option, then the above command is wrong? Should it be `ALTER SESSION SET ..`?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374451863

Review comment:
This is hard to understand. We just explained the "tables" folder. Perhaps say, "Within the tables directory, there is one file per analyzed table in the form `<storage-plugin-config-name>/<workspace-name>/<table-name>.whatever`.

Note that you are very likely (or should be) using the storage plugin *config* name, not the storage plugin name. That is "my-hdfs", not "dfs."

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-582332463
 
 
   I'll open a new PR with addressed CR comments soon.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371754567

Review comment:
This part came from statistics work and just adapted to be used with Metastore. The description is the same as for the corresponding statistics page.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366688559

Review comment:
Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the DFS storage plugin. (Does this work for local files?)

The first time you execute `ANALYZE TABLE` for a table, Drill will scan the entire tables (all files.) When you next issue the same command, Drill will scan only those files added since the previous run. The command will return the message (what?) if table statistics are up-to-date.

(Will drill automatically remove files that have been removed since the previous scan?

Please explain this in detail. Do we place a timestamp in the metadata? At what point do we set the timestamp? At the start or end of the run? Or, do we have a per-file timestamp? How do we detect if a table is new? Modification timestamp? Creation timestamp? What do we compare the timestamp against?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371842348

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371843143

Review comment:
Under metadata meant schema + statistics.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379521993
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
 
 Review comment:
   Thanks, good idea. I have added a section where enumerated problems that Metastore may help to solve.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371906506

Review comment:
thanks, added a link to schema provisioning docs.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371873405

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366683916

Review comment:
(Please explain the benefit. Faster? What is the cost? Less accurate estimates? Why should I use the fast approach, when the full approach?

If the command is `SAMPLE 50 PERCENT`, won't Drill still read 100% of my data, but throw away half of it? How is this helpful? Is there an option to specify to sample 50% of *files* to reduce disk I/O?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371146761
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
 
 Review comment:
   Thanks, updated with the text you have proposed. Also, added info on how this feature will affect existing Parquet table metadata cache files into the `Related Commands` section.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-582162980
 
 
   @arina, @bbevens, no harm done. As Bridget knows, I can keep reviewing docs forever.
   
   In fact, it is probably better to review the docs outside of Git: far too many messages get sent to the dev list if we do the review in Git.
   
   Maybe just do a second PR with the next round of revisions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374446888

Review comment:
Where did you get the following info? Is this a definitive list? Should it appear in a reference section? Or, is there a query that will return this info?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366684761

Review comment:
If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics. Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts.

(Drill is a multi-user system. We should certainly allow concurrent queries and metadata updates!)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374444214

Review comment:
Change to:

Limitations of the 1.17 release:

* Parquet...
* Disabled by default...

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366681022
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
 
 Review comment:
   Drill will infer schema for all columns, but gather statistics for none of the columns.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366690525

Review comment:
(Delete)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374636703

Review comment:
As said above, tables data can be stored any where, it have no affect on the Metastore.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366692083

Review comment:
(How can the user set this? There are no commands to do so. How would this be used? If I do a query on a file without the column, is this the value used?

Since this is the `COLUMNS` table, the column name is unfortunate: `COLUMNS.COLUMN_DEFAULT`. Better would be `DEFAULT_VALUE`. Same is true of `COLUMN_FORMAT`.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366694781

Review comment:
(This doesn't make much sense. How can a partition be in a different catalog than the table? Shouldn't this be a table attribute, not a partition attribute?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366696671

Review comment:
(Unnecessary and fragile. Computable from the table location and the partition keys.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374431849

Review comment:
in *the*

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366690683

Review comment:
`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.

(Looking at the `COLUMNS` table, it seems that this also would include "provided schema" information stored in files. True? If so, how do we know where to look for those files since they live in the root folder of each table?

This raises another question. Suppose I first create a provided schema using that file. Then I analyze the table. How is the information combined? Is it? Will Drill use the Metastore schema some times and the provided schema at other times?

Or, should I use only one or the other approaches for any one table, but not both? Maybe the provided schema provides the schema (so Drill does not infer it), while the analysis gathers column stats?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366695244

Review comment:
(These are table attributes. Neither seem like a primary key. We need a primary key that uniquely relates to the `TABLES` table. The only reasonable key is `LOCATION`. We called the path `LOCATION` in `TABLES`. The DB convention is to use the same name for the foreign key.

And, if we want an (ambiguous) short table name, it should be in the `TABLES` table, not here.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374436784

Review comment:
and --> or

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366684109

Review comment:
Related Session/System Options

The following options are set via `ALTER SYSTEM SET` or via the Drill Web console. Note, because these options affect all users, use `ALTER SYSTEM` to set them, not `ALTER SESSION`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371755721

Review comment:
Thanks, added.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373536691

Review comment:
Good clarification, thanks, added.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374450076

Review comment:
Need a bit more background.

Iceberg is essentially a file system within a file. The Iceberg file is stored in a file system. In the tutorial, we stored the metastore in your local file system. Drill is a distributed query engine, so production deployments store the metastore in your DFS such as HDFS.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371796205

Review comment:
Yes, that's correct where we use it. Added point when it may be disabled.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366682275
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
 
 Review comment:
   (This command seems to infer schema. The command is aware of partitions. Drill has a long-standing usability issue that users must do their own partition coding. If I want data from 2018-11 to 2019-02 (one quarter worth of data), I have to write the very ugly
   
   ```
   WHERE (dir0 = 2018 AND dir1 >= 11)
           OR (dir0 = 2019 AND dir1 <= 1)
   ```
   
   With Hive I can just write:
   
   ```
   WHERE transDate IN ('2018-11-01', '2019-01-31')
   ```
   Does this feature have a way to infer a column name for directory names? To know how a directory maps to a date column? If so, we should explain how that this is done and anything the user must do to make it work.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380270959

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373464139

Review comment:
Thanks, removed and updated the previous abstract with additional info.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371906774

Review comment:
Thanks, done.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374443350

Review comment:
Change to something like Analyzing a table

For this to make sense to the user, we need to explain the process: that there is a command, what happens the first time, what happens subsequent times.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379534144

Review comment:
Thanks, reworded.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366688842

Review comment:
Incremental analysis

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371157237
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
 
 Review comment:
   Thanks, this transmission is a good idea, updated docs with the text you have proposed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380209257

Review comment:
Thanks, reworded.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380199229

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374446401

Review comment:
...command on the lineitem table:

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366686311

Review comment:
Drill provides the `CREATE TABLE AS` commands to create new tables. This option causes Drill to gather schema and statistics for those tables automatically as they are written.

(Not CTTAS, correct? We explained above that `ANALYZE TABLE` will fail on a temp table. and the second "T" in "CTTAS" stands for "TEMPORARY".

Also, the text says "schema and/or column statistics." Why would I want to gather statistics without a schema? What does that even mean? Wouldn't the statistics represent an implied schema?

The default is `NONE`. But, we don't tell the user the other values. Please list them.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374449612
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,69 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
 
 Review comment:
   Drill's Metastore allows a variety of storage engines. Drill ships with a storage engine based on Iceberg... 
   
   Remove the configuration stuff so that the note directly follows the intro. Configuration should be the next section or two so no need to reference it here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374439368

Review comment:
The metastore provides a number of options to fit your environment. The default options are find in most cases. (True?) The options are set via ...

In general, you should set the options via ALTER SYSTEM so that they take effect for all users. Setting options at the session level is an advanced topic."

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374446334

Review comment:
Performing --> Perform

Since this is a tutorial.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371763891

Review comment:
Metadata keyword was added here to distinguish commands connected with metastore usage, i.e. enabling, retry_attempts.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374433097

Review comment:
Maybe change this from descriptive to proscriptive: what should the user do?

"To configure the metastore, copy the $DRILL_HOME/conf/...defaultconf file to $DRILL_HOME/conf/drill...override.conf. This is a HOCON-format file, just like drill-override.conf. Metatore properties should reside in `drill.metastore` namespace."

Then, walk though the properties one-by-one.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366697059
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/009-analyze-table-compute-statistics.md
 ##########
 @@ -99,7 +99,7 @@ Controls the 'compression' factor for the TDigest algorithm used for histogram s
 
 ## Reserved Keywords
 
-The ANALYZE TABLE statement introduces the following reserved keywords:  
+The ANALYZE TABLE COMPUTE STATISTICS statement introduces the following reserved keywords:  
 
 Review comment:
   (IMHO, adding two long words to the command did not help usability.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374445155

Review comment:
Shouldn't we explain how to set up the required storage plugin config?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374442972

Review comment:
Move the Drill 1.17 information to the Limitations section.

The first time/second time repeats the information that just appeared in the previous section.

Then, this section will be empty and can be removed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366684406

Review comment:
Same options as the *level* option above.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366685432

Review comment:
The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. (Is that where we use it?) Disable this option if Drill has inferred the schema incorrectly, or if new files arrive that have columns not in the schema. (I made up this last bit to explain why the user would care about this option. Please insert the real reason.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374445990

Review comment:
Run *the* ANALYZE TABLE command on the lineitem table:

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374448416
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
 
 Review comment:
   Dropping --> Drop
   
   Can't tell if we are still in the tutorial. If so, say "Once we are done exploring metadata we can drop the metadata for the lineitem table."
   
   We probably should have a section that summarizes the ANALYZE TABLE commands and what they do:
   
   * Add a new table with ...
   * When table data (but not schema) changes, do...
   * When the table schema changes, do...
   * If partitions are added or removed, do...
   * When a table is removed, do...
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374637017

Review comment:
SET is equivalent for ALTER SESSION SET

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371413095
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
 
 Review comment:
   The solution to use it as VARCHAR literal was done to be consistent with other usages. For example, the default level may be set to the `metastore.metadata.store.depth_level` session option:
   ```
   set `metastore.metadata.store.depth_level`='segment';
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379543754

Review comment:
Thanks, agree that it may seem a little bit confusing, so changed as you have proposed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366696537

Review comment:
(How do we know this column name? Did we make it up? Do we expect the user to use it in queries rather than the "real name"?

Database normalization rules would say that we split this into multiple tables: one that defines the partition level, another that lists all the keys (and metadata) for that level.

```
WHERE part_varchar='2019' AND `part_int`=1
```

Instead of:
```
WHERE orderDate IN ('2019-01-01', '2019-01-31')
```
)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374445819

Review comment:
Where do I do this command? From Sqlline? Do we have a SET command? Or, should this be ALTER SESSION SET?

Does this take effect for all users or just this one session? Explain: "The above command enables the metastore for just this one session."

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373450013

Review comment:
This partially repeats info form previous abstract.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374455079

Review comment:
Maybe we should take a set back. Sounds like Iceberg maintains multiple versions of each file stored in Iceberg. This allows Drill to read a prior version of the file while ANALYZE TABLE creates a new version (true? Do we have such synchronization? If not, it is a bug.)

Similarly, if a file is delete, prior versions remain.

Therefore Iceberg requires that Drill periodically remove old file versions. Drill does this on a time interval defined by ....

Note that none of the above ensure atomic operation: we can always come up with a scenario in which updates happen faster than readers. This is a bug that needs fixing.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371758903

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374636316

Review comment:
Just to be clear, Metastore can work with files from local fs, dfs, S3, it does not matters where tables data is stored, what is matters when user is using Iceberg Metastore, fs where Metastore metadata will be stored, for Iceberg preferred fs is the one that supports atomic rename, i.e. dfs, otherwise there can be concurrency issues. When other Metastore implementations will be available, for example, RDBMS Metastore user would have to have DB setup.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366687593

Review comment:
(Is `workspace.` correct? A workspace is something unique to a session. Do we mean `plugin.schema` instead?

Also, what does happen if the user works via a workspace? Does Drill work out the underlying plugin and schema name?

What happens if the user analyzes a table, then removes or changes the plugin or schema? Does the data remain in the Metastore? Can the user remove it, given that the plugin and schema names are now invalid?

Sadly, altering a plugin or schema is not done with a SQL command, so there is no way to prevent changes while metadata is present, or trigger metadata removal.

How does the user clean up in this case?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] bbevens merged pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

bbevens merged pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366679501
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
 
 Review comment:
   (Capitalize The but not "Stored". That is, it should read: "metadata. The stored schema...")

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379573563

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374442840

Review comment:
(Will Drill delete the old information? Or, will we have two entries?)

Remember, we're telling users how to use Drill to solve their problems. Might be better to say: Table schemas evolve over time. If your table adds (or removes) columns. run ANALYZE TABLE with the new set of columns. Drill will perform a full table analysis.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373564880

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366694613

Review comment:
A table can be divided into directories, called "partitions". The `PARTITION` table contains an entry for each directory.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373539540

Review comment:
Thanks.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380241423
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
 
 Review comment:
   Great idea! Added a section with the list of these items and corresponding commands.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-582101052
 
 
   @bbevens that for merging the changes but I don't think @paul-rogers has finished the review. Vova has addressed code review changes but Paul could not a chance to review the changes...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366680438
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
 
 Review comment:
   (Not clear here. If this command infers schema, do I specify here the only columns that Drill should infer? Does Drill then not store schema information for the other columns?
   
   Or, is this the list of columns for which Drill will gather statistics? Maybe Drill infers schema for all columns?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366691842

Review comment:
The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the `ANALYZE TABLE` statement appear in this table. (Correct?)

(The list of columns has no foreign key into the `TABLES` table. How would I join the `COLUMNS` table to the `TABLES` table without a foreign key?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374444478

Review comment:
Where? Does the Metastore work on the local file system? Or, only on a DFS?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374448820
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
+
+Table metadata may be dropped using `ANALYZE TABLE DROP METADATA` command:
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
++------+----------------------------------------+
+|  ok  |                summary                 |
++------+----------------------------------------+
+| true | Metadata for table [lineitem] dropped. |
++------+----------------------------------------+
+1 row selected (0.291 seconds)
+```
+
+### Collecting metadata for specific set of columns
+
+For the case when metadata for several columns should be computed and stored into the Metastore, the following command may be used:
+
+```
+apache drill (information_schema)> ANALYZE TABLE dfs.tmp.lineitem COLUMNS(l_orderkey, l_partkey) REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (94.1 seconds)
+```
+
+Now, check, that metadata is collected only for specified columns (`MIN_VAL`, `MAX_VAL`, `NDV`, etc.), but all
+ columns are present:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1      | s2      | null      | null              | false     |
 
 Review comment:
   List only two rows, then show "..." or some such. No need to show all the data.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366689869

Review comment:
Drill performs incremental analysis only when the `ANALYZE TABLE` command is identical to the previous command:

* The set of columns in the `COLUMNS` clause must be the same as the previous run.
* The metadata level in the `LEVEL` clause must be the same as the previous run.

If either of these two clauses changes, Drill will perform a full analysis over the entire table.

(Doesn't this seem to suggest that the details should be stored in the table metadata in the Metastore so the user does not have to keep track of all these details? Maybe two commands:

```
ANALYZE DATA CREATE METADATA [full options]
```
To specify the options and run a full update.

```
ANALYZE TABLE REFRESH METADATA table-name
```

To refresh metadata using the previously-specified set of columns, level, sampling ratio and so on.

Otherwise, usability is pretty awful. Impala, for example, only requires a `REFRESH` statement; the user does not have to know all the details.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371843593

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371857741

Review comment:
Thanks.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373450255

Review comment:
```suggestion
how new Metastore components like udfs, storage plugins, etc. can be added in future.
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374643271

Review comment:
As long as Iceberg is used on fs that supports atomic renames, all concurrency stuff is handled by Iceberg.

> Therefore Iceberg requires that Drill periodically remove old file versions. Drill does this on a time interval defined by ....

Yes, all Drill is need to do is just to define the interval, everything else is handled by Iceberg library internally.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374641813

Review comment:
Agree, this is DEV info, not related to users, let's remove this.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374450703

Review comment:
Where is this parameter set? As a SYSTEM/SESSION option? Actually in the config file, right? We should say so.

This default is defined in the default file, correct? If so, then "The location of the metastore file is set in the metastore....conf file. The default settings locate the file in ...

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380164420

Review comment:
Thanks, moved it to another section and removed duplicates.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380219490
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
+
+Table metadata may be dropped using `ANALYZE TABLE DROP METADATA` command:
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
++------+----------------------------------------+
+|  ok  |                summary                 |
++------+----------------------------------------+
+| true | Metadata for table [lineitem] dropped. |
++------+----------------------------------------+
+1 row selected (0.291 seconds)
+```
+
+### Collecting metadata for specific set of columns
+
+For the case when metadata for several columns should be computed and stored into the Metastore, the following command may be used:
+
+```
+apache drill (information_schema)> ANALYZE TABLE dfs.tmp.lineitem COLUMNS(l_orderkey, l_partkey) REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (94.1 seconds)
+```
+
+Now, check, that metadata is collected only for specified columns (`MIN_VAL`, `MAX_VAL`, `NDV`, etc.), but all
+ columns are present:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+---------+---------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1      | s2      | null      | null              | false     |
 
 Review comment:
   Thanks, removed excessive data.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366692691

Review comment:
The number of rows which contain nulls for this column.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371873784

Review comment:
Thanks, updated.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373566746
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,63 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date:
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For
 
 Review comment:
   Thanks, added.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366683160

Review comment:
(Please explain what `planner.enable_statistics` means.)

You cannot use this command if statistics is disabled. Statistics usage is enabled by setting the SYSTEM/SESSION option `planner.enable_statistics`. It is (what?) by default. To enable statistics:

```
ALTER SYSTEM SET `planner.enable_statistics` = true
```

(Note: This behavior seems odd. Why can't I first gather stats and review them for quality before I have my users start using them? As it is, there is no ability to gather them, enable the option for a session for testing, verify that things work right, then turn it on for everyone.

Also, what happens if I turn the option on, gather stats, then turn the option off? If we refuse to gather stats, do we automatically delete them if the option is disabled?)
if statistics usage is disabled, the command

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371907634

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371891854

Review comment:
`COLUMNS` table contains schema stored in Metastore. In [DRILL-7477](https://issues.apache.org/jira/browse/DRILL-7477) will be added possibility to use provided schema when analyzing the table. In this case, this schema will be stored in Metastore.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371682050

Review comment:
@paul-rogers, no, it doesn't infer a column name for directory names. But for the case when table has `transDate` field, which has the same values within a partition, it will be also pruned during planning.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366679238
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
 
 Review comment:
   (Here it is not clear if the command gathers scheme and statistics? Or, only schema by default?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374638553
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md
 ##########
 @@ -0,0 +1,69 @@
+---
+title: "Drill Iceberg Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For Drill 1.17,
+ this is default Drill Metastore implementation. For details on how to configure Iceberg Metastore implementation and
+ its option descriptions, please refer to [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md).
+
+{% include startnote.html %}
+Iceberg table supports concurrent writes and transactions but they are only effective on file systems that support
+ atomic rename.
+If the file system does not support atomic rename, it could lead to inconsistencies during concurrent writes.
 
 Review comment:
   User can use Iceberg Metastore on local fs or S3 but he should be aware of concurrency issues since none of them supports atomic rename, as long as one user uses it, it will be fine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366686894

Review comment:
(Again, why would I keep stats but remove metadata? This seems line an unnecessary complexity. Seems the valid option would be:

`DROP` (all)
`DROP STATISTICS` (keep the schema, clear the statistics)
)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371407625
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
 
 Review comment:
   Thanks, fixed "Stored".
   
   Yes, it will infer the schema for all columns and will collect statistics only for specified ones. replaced metadata with statistics.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374434455

Review comment:
Seems awkward to call these "components." Maybe:

"The Metastore contains many kinds of metadata. It can describe table schema and statistics. Future releases will support views. Future versions may also store UDFs and storage plugins."

However, in general, it is best not to put futures in the documentation. Better to say, "The Drill 1.16 version of the Metastore stores metadata about tables: the table schema and table statistics." Then, don't promise future work. We could say, "The Metastore is an active subproject of Drill, See (what) for more information."

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371921044

Review comment:
Field name preserves parent structs names.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373531286

Review comment:
Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373531743

Review comment:
Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374452765

Review comment:
This would be clearer with a concrete example. If you ran through the tutorial, Metastore files are stored in (where.) If you inspect this directory, you will see the following directories "dfs/tmp/lineitem".

Note: if this example is accurate, we've got a problem. "dfs" is the plugin name. I can have multiple such plugins, each of which may have a "tmp" workspace. Metastore can't tell them apart. If this is true, let's file a bug and get this fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380244476

Review comment:
Thanks, added.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379556928

Review comment:
Yes, we have a section below with the real tables and examples of how to discover metastore metadata.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373564314

Review comment:
Thanks, done.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379567240

Review comment:
Metastore supports single files also.
Added part of the info you have proposed and added references to the examples, where was described how to query partitions and segments metadata.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379569561

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380164542

Review comment:
Thanks, moved.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366690508

Review comment:
The Metastore infers the schema of your tables. You can then query that schema using the `INFORMATION_SCHEMA` tables including:

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371158678
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
 
 Review comment:
   Regarding all these ambiguities, currently, the behavior is the same as during regular select. Added a sentence to reflect this. Regarding tuning the schema, there is [DRILL-7477](https://issues.apache.org/jira/browse/DRILL-7477) Jira which will allow using the power of schema provisioning feature during collecting metadata.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366687055

Review comment:
The command will fail if the table does not exist in the Metastore. Include the `IF EXISTS` clause to ignore a missing table.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371167524
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
 
 Review comment:
   Before the `Syntax` section, was added a sentence that this command gathers schema and MIN / MAX and NULLS COUNT info named in this doc as "metadata". Also, added a sentence there regarding behavior with statistics - it will be collected only if `planner.statistics.use` option is enabled.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373453344

Review comment:
```suggestion
- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373454027

Review comment:
beta?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366683344
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
+If statistics usage is disabled (`planner.enable_statistics` is set to `false`), an error will be thrown when this clause is specified.
+
+*ESTIMATE*
+Generates estimated statistics for the table to be stored into the Metastore. Currently is not supported.
+
+*(column1, column2, ...)*
+The name of the column(s) for which Drill will generate statistics.
 
 Review comment:
   generate --> compute

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380218259
 
 

 ##########
 File path: _docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##########
 @@ -0,0 +1,408 @@
+---
+title: "Using Drill Metastore"
+parent: "Drill Metastore"
+date: 2020-01-31
+---
+
+Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
+
+The Metastore is a Beta feature; it is subject to change. We encourage you to try it and provide feedback.
+Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+## Enabling Drill Metastore
+
+To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
+
+	SET `metastore.enabled` = true;
+	ALTER SYSTEM SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Computing and storing table metadata to Drill Metastore
+
+Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table
+ has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have
+ Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves
+ query performance. In general, large tables benefit from statistics more than small tables do.
+
+Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table 
+ in the same way as it is done during regular select and computes some metadata like `MIN` / `MAX` column values and
+ `NULLS_COUNT` designated as "metadata" to be able to produce more optimizations like filter push-down, etc. If
+ `planner.statistics.use` option is enabled, this command will also calculate and store table statistics into Drill
+ Metastore.
+
+## Configuration
+
+Default Metastore configuration is defined in `drill-metastore-default.conf` file.
+It can be overridden in `drill-metastore-override.conf`. Distribution configuration can be
+indicated in `drill-metastore-distrib.conf`.
+
+All configuration properties should reside in `drill.metastore` namespace.
+Metastore implementation based on class implementation config property `drill.metastore.implementation.class`.
+The default value is the following:
+
+```
+drill.metastore: {
+  implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
+}
+```
+
+Note, that currently out of box Iceberg Metastore is available and is the default one. Though any custom
+ implementation can be added by placing the JAR into classpath which has the implementation of
+ `org.apache.drill.metastore.Metastore` interface and indicating custom class in the `drill.metastore.implementation.class`.
+
+### Metastore Components
+
+Metastore can store metadata for various components: tables, views, etc.
+Current implementation provides fully functioning support for tables component.
+Views component support is not implemented but contains stub methods to show
+how new Metastore components like UDFs, storage plugins, etc. can be added in the future.
+
+### Metastore Tables
+
+Metastore Tables component contains metadata about Drill tables, including general information, as well as
+information about table segments, files, row groups, partitions.
+
+Full table metadata consists of two major concepts: general information and top-level segments metadata.
+Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
+
+A table can be non-partitioned and partitioned. Non-partitioned tables have only one top-level segment 
+which is called default (`MetadataInfo#DEFAULT_SEGMENT_KEY`). Partitioned tables may have several top-level segments.
+Each top-level segment can include metadata about inner segments, files, row groups, and partitions.
+
+A unique table identifier in Metastore Tables is a combination of storage plugin, workspace, and table name.
+Table metadata inside is grouped by top-level segments, unique identifier of the top-level segment and its metadata
+is storage plugin, workspace, table name, and metadata key.
+
+### Related Session/System Options
+
+The following options are set via `ALTER SYSTEM SET`, or `ALTER SESSION SET` or via the Drill Web console.
+
+- **metastore.enabled**
+Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
+ to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is `false`.
+- **metastore.metadata.store.depth_level**
+Specifies the maximum level of depth for collecting metadata. Same options as the _level_ option above. Default is `'ALL'`.
+- **metastore.retrieval.retry_attempts**
+If you run the `ANALYZE TABLE` command at the same time as queries run, then the query can read incorrect or corrupt statistics.
+Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is `5`.
+- **metastore.metadata.fallback_to_file_metadata**
+Allows using [file metadata cache]({{site.baseurl}}/docs/refresh-table-metadata) for the case when required metadata is absent in the Metastore.
+Default is `true`.
+- **metastore.metadata.use_schema**
+The `ANALYZE TABLE` command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. 
+Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see [CREATE OR REPLACE SCHEMA]({{site.baseurl}}/docs/create-or-replace-schema)).
+Default is `true`.
+- **metastore.metadata.use_statistics**
+Enables obtaining table and column statistics, stored in the Metastore, at the planning stage. Default is `true`.
+Enable `planner.statistics.use` to be able to use statistics during query planning.
+- **metastore.metadata.ctas.auto-collect**
+Drill provides the [`CREATE TABLE AS`]({{site.baseurl}}/docs/create-or-replace-schema) commands to create new tables.
+This option causes Drill to gather schema and statistics for those tables automatically as they are written.
+This option is not implemented for now. Possible values: `'ALL'`, `'SCHEMA'`, `'NONE'`. Default is `'NONE'`.
+- **drill.exec.storage.implicit.last_modified_time.column.label**
+Sets the implicit column name for the last modified time (`lmt`) column. Used when producing Metastore analyze. You can
+ set the last modified time column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_index.column.label**
+Sets the implicit column name for the row group index (`rgi`) column. Used when producing Metastore analyze. You can
+ set row group index column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_length.column.label**
+Sets the implicit column name for the row group length (`rgl`) column. Used when producing Metastore analyze. You can
+ set row group length column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+- **drill.exec.storage.implicit.row_group_start.column.label**
+Sets the implicit column name for the row group start (`rgs`) column. Used when producing Metastore analyze. You can
+ set row group start column name to custom name when current column name clashes which column name present in the
+ table. If your table contains a column name with the same name as an implicit column, the implicit column takes
+ priority and shadows column from the table.
+
+## Incremental analysis
+
+If you have computed statistics for a table, and issue `ANALYZE TABLE` a second time, Drill will attempt to update
+ statistics, called "incremental analysis."
+Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse
+ actual metadata from the Metastore where possible.
+
+Drill performs incremental analysis only when the `ANALYZE TABLE command` is identical to the previous command:
+- The list of columns in the `COLUMNS` clause is a subset of interesting columns from the previous run.
+- The metadata level in the LEVEL clause must be the same as the previous run.
+
+If either of these two conditions is false, Drill will perform a full analysis over the entire table.
+
+## General Information
+
+- Drill 1.17 supports the Metastore and `ANALYZE TABLE` only for tables stored as Parquet files and only when stored in the `DFS` storage plugin.
+- The first time you execute ANALYZE TABLE for a table, Drill will scan the entire table (all files.)
+When you issue the same command for the next time, Drill will scan only those files added since the previous run.
+The command will return the following message if table statistics are up-to-date:
+
+
+```
+apache drill (dfs.tmp)> analyze table lineitem refresh metadata;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+```
+
+### Metadata usage
+
+Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
+ Drill will use statistics to "prune" the scan. That is, to identify those directories, files or row groups that
+ do not contain data that your query needs. If you add new files or directories and do not rerun `ANALYZE TABLE`,
+ then Drill will assume that existing metadata is invalid and wouldn't use it. Periodically rerun `ANALYZE TABLE` so
+ that Drill can use table metadata when possible.
+
+### Limitations
+
+This feature is currently in the beta phase (preview, experimental) for Drill 1.17 and only applies to Parquet
+ tables in this release. You must enable this feature through the `metastore.enabled` system/session option.
+
+## Examples
+
+Examples throughout this topic use the files and directories described in the following section, Directory, and File Setup.
+
+### Directory and File Setup
+
+Download [TPC-H sf1 tables](https://s3-us-west-1.amazonaws.com/drill-public/tpch/sf1/tpch_sf1_parquet.tar.gz) and
+ unpack archive.
+
+Create lineitem directory in `/tmp/` and two subdirectories under `/tmp/lineitem` named `s1` and `s2` and copy there table data:
+
+    mkdir /tmp/lineitem
+    mkdir /tmp/lineitem/s1
+    mkdir /tmp/lineitem/s2
+    cp TPCH/lineitem /tmp/lineitem/s1
+    cp TPCH/lineitem /tmp/lineitem/s2
+
+Query the directory `/tmp/lineitem`:
+
+```
+SELECT count(*) FROM dfs.tmp.lineitem;
++----------+
+|  EXPR$0  |
++----------+
+| 12002430 |
++----------+
+1 row selected (0.291 seconds)
+```
+
+Notice that the query plan contains a group scan with `usedMetastore = false`:
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
+00-02        DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
+```
+
+### Computing and storing table metadata to Drill Metastore
+
+Enable Drill Metastore:
+
+```
+SET `metastore.enabled` = true;
+```
+
+Run [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table, whose metadata should
+ be computed and stored into the Drill Metastore:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++------+-------------------------------------------------------------+
+|  ok  |                           summary                           |
++------+-------------------------------------------------------------+
+| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
++------+-------------------------------------------------------------+
+1 row selected (32.257 seconds)
+```
+
+The output of this command provides the status of the command execution and its summary.
+
+Once, table metadata is collected and stored in the Metastore, it will be used when querying the table. To ensure that it was used, its
+ info was added to the group scan (`usedMetastore=true` entry in `ParquetGroupScan`):
+
+
+```
+00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
+00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
+00-02        DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
+```
+
+### Performing incremental analysis
+
+Rerun [ANALYZE TABLE]({{site.baseurl}}/docs/analyze-table-refresh-metadata) command on the table with previously collected metadata:
+
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
++-------+---------------------------------------------------------+
+|  ok   |                         summary                         |
++-------+---------------------------------------------------------+
+| false | Table metadata is up to date, analyze wasn't performed. |
++-------+---------------------------------------------------------+
+1 row selected (0.249 seconds)
+```
+
+### Exposing Drill Metastore metadata through `INFORMATION_SCHEMA` tables
+
+Drill exposes some Metastore tables metadata through `INFORMATION_SCHEMA` tables.
+Note, that Metastore metadata will be exposed to the info schema, only if Metastore is enabled. If it is disabled, info
+ tables won't contain Metastore metadata.
+
+`TABLES` table includes the set of tables on which you have run `ANALYZE TABLE`.
+Description of Metastore-specific columns:
+
+|Column name            |Type       |Nullable   |Description                                                                                            |
+|-----------------------|-----------|-----------|-------------------------------------------------------------------------------------------------------|
+|`TABLE_SOURCE`         |VARCHAR    |YES        |Table data type: `PARQUET`, `CSV`, `JSON`                                                              |
+|`LOCATION`             |VARCHAR    |YES        |Table location: `/tmp/nation`                                                                          |
+|`NUM_ROWS`             |BIGINT     |YES        |The total number of rows in all files of the table. Null if not known                                  |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Timestamp of the most-recently modified file within the table. Updated on each `ANALYZE TABLE` run.    |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE |   LOCATION    | NUM_ROWS |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | TABLE      | PARQUET      | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+------------+--------------+---------------+----------+-----------------------+
+1 row selected (0.157 seconds)
+```
+
+The `COLUMNS` table describes the columns within each table. Only those columns listed in the `COLUMNS` clause of the
+ `ANALYZE TABLE` statement appear in this table.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`COLUMN_DEFAULT`       |VARCHAR    |YES        |Column default value.                                                                                                                                                              |
+|`COLUMN_FORMAT`        |VARCHAR    |YES        |Usually applicable for date time columns: `yyyy-MM-dd`. See [Format for Date, Time Conversion]({{site.baseurl}}/docs/create-or-replace-schema/#format-for-date-time-conversion).   |
+|`NUM_NULLS`            |BIGINT     |YES        |The number of rows which contain nulls for this column.                                                                                                                            |
+|`MIN_VAL`              |VARCHAR    |YES        |The minimum value of the column expressed as a string. For example: `'-273'`.                                                                                                      |
+|`MAX_VAL`              |VARCHAR    |YES        |The maximum value of the column expressed as a string. For example: `'100500'`.                                                                                                    |
+|`NDV`                  |FLOAT8     |YES        |Number of distinct values in column, expressed in Double.                                                                                                                          |
+|`EST_NUM_NON_NULLS`    |FLOAT8     |YES        |Estimated number of non null values, expressed in Double.                                                                                                                          |
+|`IS_NESTED`            |BIT        |NO         |If column is nested. Nested columns are extracted from columns with struct type.                                                                                                   |
+
+Example of its content:
+
+```
+apache drill> SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME |   COLUMN_NAME   | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |     DATA_TYPE     | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS |   MIN_VAL    |                   MAX_VAL                   |    NDV    | EST_NUM_NON_NULLS | IS_NESTED |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+| DRILL         | dfs.tmp      | lineitem   | dir0            | 1                | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | s1           | s2                                          | null      | null              | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_orderkey      | 2                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 6000000                                     | 1499876.0 | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_partkey       | 3                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 200000                                      | 199857.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_suppkey       | 4                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 10000                                       | 10001.0   | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linenumber    | 5                | null           | YES         | INTEGER           | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 11          | null          | 0         | 1            | 7                                           | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_quantity      | 6                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 1.0          | 50.0                                        | 50.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_extendedprice | 7                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 901.0        | 104949.5                                    | 933142.0  | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_discount      | 8                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.1                                         | 11.0      | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_tax           | 9                | null           | YES         | DOUBLE            | null                     | null                   | 0                 | 2                       | 0             | null               | null          | null               | 24          | null          | 0         | 0.0          | 0.08                                        | 9.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_returnflag    | 10               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | A            | R                                           | 3.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_linestatus    | 11               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | F            | O                                           | 2.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipdate      | 12               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694310400000 | 912470400000                                | 2526.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_commitdate    | 13               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 696816000000 | 909792000000                                | 2466.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_receiptdate   | 14               | null           | YES         | DATE              | null                     | null                   | null              | null                    | null          | 10                 | null          | null               | 10          | null          | 0         | 694483200000 | 915062400000                                | 2554.0    | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipinstruct  | 15               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | COLLECT COD  | TAKE BACK RETURN                            | 4.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_shipmode      | 16               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         | AIR          | TRUCK                                       | 7.0       | 1.200243E7        | false     |
+| DRILL         | dfs.tmp      | lineitem   | l_comment       | 17               | null           | YES         | CHARACTER VARYING | 65535                    | 65535                  | null              | null                    | null          | null               | null          | null               | 65535       | null          | 0         |  Tiresias    | zzle? slyly final platelets sleep quickly.  | 4586320.0 | 1.200243E7        | false     |
++---------------+--------------+------------+-----------------+------------------+----------------+-------------+-------------------+--------------------------+------------------------+-------------------+-------------------------+---------------+--------------------+---------------+--------------------+-------------+---------------+-----------+--------------+---------------------------------------------+-----------+-------------------+-----------+
+17 rows selected (0.187 seconds)
+```
+
+A table can be divided into directories, called "partitions". The `PARTITIONS` table contains an entry for each directory.
+
+|Column name            |Type       |Nullable   |Description                                                                                                                                                                        |
+|-----------------------|-----------|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|`TABLE_CATALOG`        |VARCHAR    |YES        |Table catalog (currently we have only one catalog): `DRILL`.                                                                                                                       |
+|`TABLE_SCHEMA`         |VARCHAR    |YES        |Table schema: `dfs.tmp`.                                                                                                                                                           |
+|`TABLE_NAME`           |VARCHAR    |YES        |Table name: `nation`.                                                                                                                                                              |
+|`METADATA_KEY`         |VARCHAR    |YES        |Top level segment key, the same for all nested segments and partitions: `part_int=3`.                                                                                              |
+|`METADATA_TYPE`        |VARCHAR    |YES        |`SEGMENT` or `PARTITION`. Partition here corresponds to "Drill partition", though segment corresponds to data parts like partitions in general case, for example, Hive partition.  |
+|`METADATA_IDENTIFIER`  |VARCHAR    |YES        |Current metadata identifier: `part_int=3/part_varchar=g`. It is unique value for segment or partition within the table.                                                            |
+|`PARTITION_COLUMN`     |VARCHAR    |YES        |Partition column name: `part_varchar`.                                                                                                                                             |
+|`PARTITION_VALUE`      |VARCHAR    |YES        |Partition column value: `g`.                                                                                                                                                       |
+|`LOCATION`             |VARCHAR    |YES        |Segment location, `null` for partitions: `/tmp/nation/part_int=3`.                                                                                                                 |
+|`LAST_MODIFIED_TIME`   |TIMESTAMP  |YES        |Last modification time.                                                                                                                                                            |
+
+Example of its content:
+
+```
+apache drill (information_schema)> SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE |     LOCATION     |  LAST_MODIFIED_TIME   |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+| DRILL         | dfs.tmp      | lineitem   | s2           | SEGMENT       | s2                  | `dir0`           | s2              | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
+| DRILL         | dfs.tmp      | lineitem   | s1           | SEGMENT       | s1                  | `dir0`           | s1              | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
++---------------+--------------+------------+--------------+---------------+---------------------+------------------+-----------------+------------------+-----------------------+
+2 rows selected (0.149 seconds)
+```
+
+### Dropping table metadata
+
+Table metadata may be dropped using `ANALYZE TABLE DROP METADATA` command:
+
+```
+apache drill> ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
++------+----------------------------------------+
+|  ok  |                summary                 |
++------+----------------------------------------+
+| true | Metadata for table [lineitem] dropped. |
++------+----------------------------------------+
+1 row selected (0.291 seconds)
+```
+
+### Collecting metadata for specific set of columns
 
 Review comment:
   Thanks, reworded and added this sentence.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366678153
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
 
 Review comment:
   To use the Drill Metastore, you must enable it with the following command:
   
   (BTW: Why the `SET` command and not `ALTER SYSTEM`? Seems an unnecessary complexity.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-573703899
 
 
   @vvysotskyi overall good regarding analyze command but I think we need to add more info about Drill Metastore notion in general. I suggest we create separate section where we'll describe what is Drill Metastore, how to configure it, which one is used by default. Plus add examples, how all these including analyze command is used together (create table, collect metadata, show that it is present via info schema tables, show that is is used when executing the query in query plan etc).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371902565

Review comment:
There are listed only columns related to the metastore. The complete list is too long, so I've listed only metastore-related columns. Columns which will refer to a specific table were already present in the previous version of this table: `TABLE_CATALOG`, `TABLE_SCHEMA`, `TABLE_NAME`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374451335

Review comment:
"will be" --> is

Here and below.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371408775
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
 
 Review comment:
   Thanks, reworded in the way you have pointed.
   Regarding `EXCLUDE` clause, I like this idea, will create Jira to implement it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371873857

Review comment:
Done.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371745370
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
+
+*level*
+Optional varchar literal which specifies maximum level depth for collecting metadata.
+Possible values: `TABLE`, `SEGMENT`, `PARTITION`, `FILE`, `ROW_GROUP`, `ALL`. Default is `ALL`.
+
+*COMPUTE*
+Generates statistics for the table to be stored into the Metastore.
+If statistics usage is disabled (`planner.enable_statistics` is set to `false`), an error will be thrown when this clause is specified.
+
+*ESTIMATE*
+Generates estimated statistics for the table to be stored into the Metastore. Currently is not supported.
+
+*(column1, column2, ...)*
+The name of the column(s) for which Drill will generate statistics.
 
 Review comment:
   Thanks, replaced.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374637580

Review comment:
Agree we can move to separate sub-section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371167638
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
 
 Review comment:
   Good point, thanks, added.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366693623

Review comment:
(This column is redundant. By definition, this value is `TABLES`.`NUM_ROWS` - `COLUMNS`.`NUM_NULLS`. If not, the math won't work.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366679074

Review comment:
(We just told the user that a metastore exists and how to enable it. Now we tell them the syntax of commands. We need a transition.)

Once you enable the Metastore, the next step is to populate it with data. Drill can query a table whether that table has a Metastore entry or not. (If you are familiar with Hive, then you know that Hive requires that all tables have Hive Metastore entries before you can query them.) In Drill, only add data to the Metastore when doing so improves query performance. In general, large tables benefit from statistics more than small tables do.

Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table. Drill not only infers the schema, it optionally computes statistics about your table.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366677818

Review comment:
Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.

The Metastore is an Alpha feature; it is subject to change. We encourage you to try it and provide feedback. Because the Metastore is in Beta, the SQL commands and Metastore formats may change in the next release.

(Looks like this appears in Limitations below, but it should be stated right up front. Note the addition of warnings of what will change.

Please explain how this feature relates to the previous Parquet metadata feature. In particular, will enabling this feature break existing behavior?)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva edited a comment on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva edited a comment on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-573703899
 
 
   @vvysotskyi overall good regarding analyze command but I think we need to add more info about Drill Metastore notion in general. I suggest we create separate section where we'll describe what is Drill Metastore, how to configure it, which one is used by default. Plus add examples, how all these including analyze command are used together (create table, collect metadata, show that it is present via info schema tables, show that is is used when executing the query in query plan etc).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366680924
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
+
+*COLUMNS (col1, col2, ...)*
+Optional names of the column(s) for which Drill will generate and store metadata. the Stored schema will include all table columns.
+
+*COLUMNS NONE*
+Specifies to ignore collecting and storing metadata for all table columns.
 
 Review comment:
   (What is the difference between `COLUMNS NONE` and no `COLUMNS` clause at all?
   
   Might it be worth while having an `EXCLUDE` clause? If I have 12 columns, but I don't need the `$type` and `$recordNo` columns, it is easier to list the columns to exclude than to list all columns except these. This is especially true because the big benefit seems to be that Drill will infer the columns; I have to list them (but not the types), it seems we have a funny worst of both worlds situation.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-581347442
 
 
   @paul-rogers is anything else @vvysotskyi need to address? Could you please take a look?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r379543295

Review comment:
Thanks, updated section with the info you have proposed and added a link to main Jira.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366696042

Review comment:
(Please explain hat this means.)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373450401

Review comment:
```suggestion
Table general information contains basic table information and corresponds to the `BaseTableMetadata` class.
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373453026

Review comment:
Please add: You can set last modified time column name to custom name when current column name clashes which column name present in the table. If your table contains column name with the same name as implicit column, implicit column takes priority and shadows column from table.

Please add the same for similar options.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366679282
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
+
+The ANALYZE TABLE REFRESH METADATA statement supports the following syntax:
+
+	ANALYZE TABLE [table_name] [COLUMNS {(col1, col2, ...) | NONE}]
+	REFRESH METADATA ['level' LEVEL]
+	[{COMPUTE | ESTIMATE} | STATISTICS [(column1, column2, ...)]
+	[ SAMPLE number PERCENT ]]
+
+## Parameters
+
+*table_name*
+The name of the table or directory for which Drill will collect table metadata. If the table does not exist, or the table
+ is temporary, the command fails and metadata is not collected and stored.
 
 Review comment:
   or if you do not have permission to read the table

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374442403

Review comment:
This sentence touches on two topics not yet described: "compute statistics" and "ANALYZE TABLE". We need to explain that. Something like:

You create Metastore metadata by running the ANALYZE TABLE command. The first time you run it, the Metastore will infer the schema and (depending on which options you have selected), populate statistics.

Tables change over time. To keep the Metastore up-to-date, you must periodically run ANALYZE TABLE again on each changed table. When you do ... (insert material from here).

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r380209127

Review comment:
Thanks, replaced.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374435223

Review comment:
We've already given a general description about table metadata. Here we should be very specific. For example:

* Table schema. For each column: name, type, nullability, ...
* Table statistics. This itself has two categories:
* Summary statistics: MIN, MAX, NULL count.
* Detail statistics: histograms, ...

Please be very specific.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

arina-ielchiieva commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373453468

Review comment:
```suggestion
When you issue the same command for the next time, Drill will scan only those files added since the previous run.
```

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r374454183

Review comment:
Good info, but unclear. First, please explain what is meant by the Parquet file. Iceberg is a file system within a file, right? So, the user can never see the Parquet files? If so, then this section is moot: the user can't do anything with the information.

However, if Iceberg provides zip-like utilities to inspect the Iceberg file, then we can tell the user how to use them. Then we can explain what they will see.

I did not follow the file format. There must be a file for the table itself, right? That has schema, etc?

Then, there is a file for each partition? What is a "metadata key"? Is this the concatenated directory names? If I have "mytable/2016/12/01/files.parquet", will my partition key be "2016/12/01"? If so, partition keys *must* be unique: the file system demands it. However, if the key is "20161201", then the name can be ambiguous, but this is self-inflicted.

Then, where does the UUID fit in? Do we have a table from partition directory keys to UUIDs?

Since the user has no control, and relies on us to make things work, the sentence about failure can be removed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373537700

Review comment:
Thanks, fixed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r373530287

Review comment:
Thanks, provided the example with a default value and noted about that and the possibility of using custom implementation as you have suggested.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

vvysotskyi commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r371857407

Review comment:
Yes, it works for files on the local file system. Regarding "return message", `ANALYZE TABLE` as other commands (like create table, ect.), returns two columns with operation status and summary. Added example of such output.
Drill will remove metadata for non-existing files within the table during incremental analyze. We store a list of table files and their last modification times into the metastore, and during producing analyze, compare lists of actual and previous files and their timestamps. More detailed it is described in this doc: https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit?usp=sharing

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [drill] bbevens commented on issue #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

bbevens commented on issue #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#issuecomment-582120502
 
 
   oh no...sorry...I thought the doc was complete. Should I remove the Iceberg Metastore topic from the site? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore

Posted by GitBox <gi...@apache.org>.

paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore
URL: https://github.com/apache/drill/pull/1953#discussion_r366680128
 
 

 ##########
 File path: _docs/sql-reference/sql-commands/007-analyze-table-refresh-metadata.md
 ##########
 @@ -0,0 +1,158 @@
+---
+title: "ANALYZE TABLE REFRESH METADATA"
+parent: "SQL Commands"
+date: 2020-01-13
+---
+
+Starting from Drill 1.17, you can store table metadata (including schema and computed statistics) into Drill Metastore.
+This metadata will be used when querying a table for more optimal plan creation.
+
+{% include startnote.html %}In Drill 1.17, this feature is supported for Parquet tables only and is disabled by default.{% include endnote.html %}
+
+To enable Drill Metastore usage, the following option `metastore.enabled` should be set to `true`, as shown:
+
+	SET `metastore.enabled` = true;
+
+Alternatively, you can enable the option in the Drill Web UI at `http://<drill-hostname-or-ip-address>:8047/options`.
+
+## Syntax
 
 Review comment:
   (If the above is true, it raises all sorts of questions. How do we handle conflicting types? Conflicting names (Foo, foo and FOO)? How does the user specify that "column X is coming, when it arrives in new files, it will be of type DECIMAL"?)
   
   Inferring schema is good, but a human must fine tune the schema to express intent, especially when the information is ambiguous. And, a main reason to have a schema is to deal with ambiguous data.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services