You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/04/17 14:23:13 UTC
[GitHub] [drill] arina-ielchiieva opened a new pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore
arina-ielchiieva opened a new pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore
URL: https://github.com/apache/drill/pull/2060
# [DRILL-7706](https://issues.apache.org/jira/browse/DRILL-7706): Implement Drill RDBMS Metastore
## Description
Currently Drill has only one Metastore implementation based on Iceberg tables. Iceberg tables are file based storage that supports concurrent writes / reads but required to be placed on distributed file system.
This PR aims to implement Drill RDBMS Metastore which will store Drill Metastore metadata in the database of the user's choice. Currently, PostgreSQL and MySQL databases are supported, others might work as well but no testing was done. Also out of box for demonstration / testing purposes Drill will setup SQLite file based embedded database but this is only applicable for Drill in embedded mode.
1. Fix issue with undeterministic execution of batch update / delete statements, now they will be executed in the same order as they were added.
2. Abstracted Metastore common test classes to be used by different Metastore implementations.
3. Added drill-metastore-override-example.conf with example of Drill Metastore configuration.
4. Replaced list of metadata types which are required to be passed during read / write operations with set to avoid possible duplicates.
5. Add RDBMS Metastore implementation, README.md and unit tests.
## Documentation
RDBMS Metastore section should be added: http://drill.apache.org/docs/using-drill-metastore/
## Testing
Ran all unit tests, Functional & Advanced. Tested manually with SQLite, PostgreSQL, MySQL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on issue #2060: DRILL-7706: Implement
Drill RDBMS Metastore
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #2060: DRILL-7706: Implement Drill RDBMS Metastore
URL: https://github.com/apache/drill/pull/2060#issuecomment-615395578
@arina-ielchiieva, another related question. We currently use ZK to store things like plugin configs, dynamic UDFS, etc. Charles just asked about the problems storing credentials that way.
I wonder if the DB support here is (or can be) generalized to allow a variety of stores (the way we have multiple "pstores" for ZK.) If so, then, over time, we could implement alternative, DB-based "pstores" for plugins, UDFs, security credentials and more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410823535
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
Review comment:
SQLite is an embedded database; is not distributed. SQLite is good for trying out the feature, for testing, for a running Drill in embedded mode, and perhaps for a single-node Drill "cluster". If should not be used in a multi-node cluster. Each Drillbit will have its own version of the truth and behavior will be undefined and incorrect.
((Question: can we detect that more than one Drillbit is running and fail or log an error if using Sqllite?))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883326
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410885212
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
Review comment:
Agree, but it was used as unique path identifier plus in config we might not have access to the variables defined in shell scripts. So, I guess for embedded mode it's ok, plus the same approach used for Dynamic UDFs and Iceberg Metastore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410827194
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
Review comment:
~~It~~ Liquibase converts **the** yaml ~~content~~ **specification** into **the** DDL ~~/~~ **and** DML commands ~~suitable to database syntax to database syntax based on connection details.~~ **required for the configured database.**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410730382
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: OWNER
+ type: VARCHAR(100)
+ - column:
+ name: TABLE_TYPE
+ type: VARCHAR(100)
+ - column:
+ name: METADATA_KEY
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: METADATA_TYPE
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: LOCATION
+ type: VARCHAR(500)
Review comment:
Realy a good choice for this value. Though some linux-based systems limit its size to 256 bytes, other systems like Windows removed the existing 260 characters limit. So I think 500 value is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410932499
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
Review comment:
1. Done.
2. Overall.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on issue #2060: DRILL-7706:
Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#issuecomment-616126481
@vvysotskyi / @paul-rogers addressed code review comments.
Explained in comments why denormalized db structure was chosen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410823749
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
Review comment:
...use **the** `com...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883955
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
Review comment:
1. Updated the comment.
2. Not, we cannot so Drill admin should know what he does.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410822887
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
Review comment:
((This is an awkward location. `zk.root` is a path in ZK, not on the file system. The location should be relative to $DRILL_CONFIG or $DRILL_SITE so that it resides in the home directory, which should be separate from the Drill install in a production system. I don't recall how to get that location in Drill, it is given by `DRILL_CONF_DIR` in the shell scripts.))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410490264
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SchemaUtilites.java
##########
@@ -218,7 +218,7 @@ private static AbstractSchema resolveToDrillSchemaInternal (SchemaPlus defaultSc
if (isRootSchema(schema)) {
throw UserException.validationError()
- .message("Root schema is immutable. Creating or dropping tables/views is not allowed in root schema." +
+ .message("Root schema is immutable. Creating or dropping tables/views is not allowed in root schema. " +
Review comment:
Maybe: "Drill does not allow creating or deleting tables or views in the root schema."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410831312
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/RdbmsMetastore.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.typesafe.config.Config;
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+import liquibase.Liquibase;
+import liquibase.database.Database;
+import liquibase.database.DatabaseFactory;
+import liquibase.database.jvm.JdbcConnection;
+import liquibase.resource.ClassLoaderResourceAccessor;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.metastore.Metastore;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.views.Views;
+import org.apache.drill.metastore.config.MetastoreConfigConstants;
+import org.apache.drill.metastore.rdbms.components.tables.RdbmsTables;
+import org.apache.drill.metastore.rdbms.config.RdbmsConfigConstants;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.jooq.SQLDialect;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.sql.DataSource;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.util.Properties;
+
+/**
+ * RDBMS Drill Metastore implementation that creates necessary tables using Liquibase,
+ * initializes data source using provided config.
+ */
+public class RdbmsMetastore implements Metastore {
+
+ private static final Logger logger = LoggerFactory.getLogger(RdbmsMetastore.class);
+
+ private static final String LIQUIBASE_CHANGELOG_FILE = "db/changelog/changelog.yaml";
+
+ private final QueryExecutorProvider executorProvider;
+
+ public RdbmsMetastore(DrillConfig config) {
+ HikariDataSource dataSource = dataSource(config);
+ this.executorProvider = new QueryExecutorProvider(dataSource);
+ initTables(dataSource);
+ }
+
+ @Override
+ public Tables tables() {
+ return new RdbmsTables(executorProvider);
+ }
+
+ @Override
+ public Views views() {
+ throw new UnsupportedOperationException("Views metadata support is not implemented");
+ }
+
+ @Override
+ public void close() {
+ executorProvider.close();
+ }
+
+ /**
+ * Prepares database before initializing data source based on its type,
+ * initializes {@link HikariDataSource} instance and configures it based on given
+ * Metastore configuration.
+ * Basic parameters such as driver, url, user name and password are set using setters.
+ * Other source parameters are set dynamically through the properties. See the list
+ * of available Hikari properties: <a href="https://github.com/brettwooldridge/HikariCP">.
+ *
+ * @param config Metastore config
+ * @return Hikari data source instance
+ * @throws RdbmsMetastoreException if unable to configure Hikari data source
+ */
+ private HikariDataSource dataSource(DrillConfig config) {
+ prepareDatabase(config);
+ try {
+ Properties properties = new Properties();
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES)) {
+ Config propertiesConfig = config.getConfig(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES);
+ propertiesConfig.entrySet().forEach(e -> properties.put(e.getKey(), e.getValue().unwrapped()));
+ }
+ HikariConfig hikariConfig = new HikariConfig(properties);
+ hikariConfig.setDriverClassName(config.getString(RdbmsConfigConstants.DATA_SOURCE_DRIVER));
+ hikariConfig.setJdbcUrl(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_USER_NAME)) {
+ hikariConfig.setUsername(config.getString(RdbmsConfigConstants.DATA_SOURCE_USER_NAME));
+ }
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PASSWORD)) {
+ hikariConfig.setPassword(config.getString(RdbmsConfigConstants.DATA_SOURCE_PASSWORD));
+ }
+ return new HikariDataSource(hikariConfig);
+ } catch (RuntimeException e) {
+ throw new RdbmsMetastoreException("Unable to init RDBMS Metastore data source: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Prepares database prior to data source configuration based on its type.
+ * For example, SQLite database requires database path to exists prior to database initialization.
+ *
+ * @param config Metastore config
+ */
+ private void prepareDatabase(DrillConfig config) {
+ SQLDialect dialect = JDBCUtils.dialect(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ switch (dialect) {
+ case SQLITE:
Review comment:
This kind of approach will get complex as we add more DBs. Suggestion: define a `DbHelper` (pick a better name) interface with calls for `prepareDatabase()`, etc. Implement the interface for each DB that needs extra work. Provide a do-nothing default implementation for other DBs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410887631
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410833801
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
Review comment:
In an RDBMS, if we have a FK into the Tables table, then do we need the column location here? Isn't the metadata type determined by the fact that this is the Columns table?
If a table is tracked within a workspace, does it have multiple entries if workspaces are hierarchical? Should we map tables relative to their plugin, with code to work out relative paths for a workspace?
The schema is a composite structure: type, nullable/array, precision, scale. For maps, there is nested structure. Should those be individual columns? Statistics are also a collection of values. Should those be columns rather than just a blob as they seem to be here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410827641
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
Review comment:
The Drill Metastore tables are created in the database schema indicated in the connection URL. This will be the default schema unless you specify a different schema. Drill will not create the schema, however. Best practice is to create a schema within your database for the Drill metastore before initializing the metastore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410932691
##########
File path: pom.xml
##########
@@ -605,7 +605,7 @@
<configuration>
<rules>
<bannedDependencies>
- <excludes>
+ <excludes>SchemaUtilites.java
Review comment:
Thanks, fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410892203
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/RdbmsTables.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.tables.TablesMetadataTypeValidator;
+import org.apache.drill.metastore.operate.Metadata;
+import org.apache.drill.metastore.operate.Modify;
+import org.apache.drill.metastore.operate.Read;
+import org.apache.drill.metastore.rdbms.QueryExecutorProvider;
+import org.apache.drill.metastore.rdbms.RdbmsMetastoreContext;
+import org.apache.drill.metastore.rdbms.operate.RdbmsMetadata;
+import org.apache.drill.metastore.rdbms.operate.RdbmsModify;
+import org.apache.drill.metastore.rdbms.operate.RdbmsRead;
+import org.apache.drill.metastore.rdbms.transform.Transformer;
+
+/**
+ * Metastore Tables component which stores tables metadata in the corresponding RDBMS tables:
+ * TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+ * Provides methods to read and modify tables metadata.
+ */
+public class RdbmsTables implements Tables, RdbmsMetastoreContext<TableMetadataUnit> {
Review comment:
`Tables` is interface defined in Metastore API. I suggest we leave it as it to avoid refactoring in other Metastore implementations. We can rename it later is we want to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410823667
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
Review comment:
((As above.))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410885499
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410831759
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/RdbmsTables.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.tables.TablesMetadataTypeValidator;
+import org.apache.drill.metastore.operate.Metadata;
+import org.apache.drill.metastore.operate.Modify;
+import org.apache.drill.metastore.operate.Read;
+import org.apache.drill.metastore.rdbms.QueryExecutorProvider;
+import org.apache.drill.metastore.rdbms.RdbmsMetastoreContext;
+import org.apache.drill.metastore.rdbms.operate.RdbmsMetadata;
+import org.apache.drill.metastore.rdbms.operate.RdbmsModify;
+import org.apache.drill.metastore.rdbms.operate.RdbmsRead;
+import org.apache.drill.metastore.rdbms.transform.Transformer;
+
+/**
+ * Metastore Tables component which stores tables metadata in the corresponding RDBMS tables:
+ * TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+ * Provides methods to read and modify tables metadata.
+ */
+public class RdbmsTables implements Tables, RdbmsMetastoreContext<TableMetadataUnit> {
Review comment:
The term "tables" in a query engine is excessively overloaded. Do we mean system tables, the physical table on disk, a Calcite representation of a table, the metadata description of a table, ...? Perhaps rename the classes `TableMetadata` or `TableDescription` (you pick a name) and `RdbmsTableWhatever`. Else, my first thought was that this is the class that will manage the tables (there is that term again) we created in the RDBMS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410824479
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
Review comment:
~~This component~~ **The `tables` component** provides...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896824
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: OWNER
+ type: VARCHAR(100)
+ - column:
+ name: TABLE_TYPE
+ type: VARCHAR(100)
+ - column:
+ name: METADATA_KEY
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: METADATA_TYPE
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: LOCATION
+ type: VARCHAR(500)
+ - column:
+ name: INTERESTING_COLUMNS
+ type: CLOB
Review comment:
Explained above regarding the choice of denormalized schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410821615
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
Review comment:
To enable **the** RDBMS Metastore**,** create **the** `drill-metastore-override.conf` **file in your config directory** and ~~indicate~~ **specify the** RDBMS Metastore class:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883280
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410826549
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
Review comment:
Table ~~s~~ creation
**The** RDBMS Metastore ~~is using~~ **uses** [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
to create ~~all~~ **the** needed tables during RDBMS Metastore initialization ~~, users~~ **. Users** should not create ~~any~~ tables manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410886908
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410821373
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
Review comment:
Currently, **the** RDBMS Metastore is not **the** default ~~Drill Metastore~~ implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883031
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on issue #2060: DRILL-7706:
Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#issuecomment-615879117
@paul-rogers good questions. Though none of them are addressed in this PR, since this PR only adds support for Drill Metastore `tables` component. I will provide below extended answer to your questions with the guidelines on what could be done to support use cases you have asked about.
`First question`:
Short answer: yes but some parts should be implemented first.
Extended answer:
Let's assume we want to store schema for HTTP plugin tables in Drill Metastore and use it when querying data from this plugin.
`ANALYZE TABLE` command collects data about table, including schema, statistics etc. It allows user to provide schema inline as well. For example: `ANALYZE TABLE table(dfs.tmp.region(schema=>'inline=(id int, country varchar)')) REFRESH METADATA`.
You can also call it only with schema and without statistics but you will need to disable statistics collection using session option: `planner.statistics.use`, in future `ANALYZE TABLE` command can be updated to do this without setting the option.
Now `ANALYZE TABLE` command works only with file based tables. So first we will need to extend it to support analysis for tables from storage plugins. Maybe add interfaces that each storage plugin would need to implement.
`ANALYZE TABLE` command will gather data for such tables and transfer it to the Drill Metastore. Drill Metastore will store it and will be able to provide it when asked (this part is implemented already).
Currently, only file based format plugins work with Drill Metastore, so last step would be to integrate Drill Metastore usage in HTTP plugin or any other plugin.
`Second question`:
Short answer: yes, but you will have to implement new components.
Extended answer:
Drill Metastore consists of megastore-api (which contains Metastore interfaces and general classes) and metastore implementations, now we have Iceberg, this PR adds also RDBMS.
Drill Metastore interface consists of components. Now we have only `tables` component which stores metadata for Drill tables, including their segments, files, row groups and partitions if any. `Views` component is present but not implemented. https://github.com/apache/drill/blob/master/metastore/metastore-api/src/main/java/org/apache/drill/metastore/Metastore.java
So what if you want to add new component, for example, `pstore`? Just add new component to the `DrillMetastore` interface. As you wrote, it would store
information `for plugins, UDFs, security credentials and more` so I think it's better to create separate component to each information type:
```
Plugins plugins();
Udfs udfs();
Credentials credentials();
```
For each component you would also need to come up with some `unit` which will be used to provide info to the Metastore and back. For example, for `tables` component there is `TableMetadataUnit` unit.
Then you would need to implement these interfaces in Drill Iceberg and RDBMS Metastore implementations. In Iceberg each component would have it's own Iceberg table, in RDBMS - one or several database tables. Most of the code is already written, you would just need to add code specific for each new component.
And last step is to integrate Metastore calls in Drill code where you will need to use it. `DrillMetastore` is accessible though `DrillbitContext`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896698
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: OWNER
+ type: VARCHAR(100)
+ - column:
+ name: TABLE_TYPE
+ type: VARCHAR(100)
+ - column:
+ name: METADATA_KEY
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: METADATA_TYPE
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: LOCATION
+ type: VARCHAR(500)
Review comment:
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410888817
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
+
+JOOQ generates SQL statements based on SQL dialect determined by database connection details.
+List of supported dialects: https://www.jooq.org/javadoc/3.13.x/org.jooq/org/jooq/SQLDialect.html.
+Note: dialects annotated with `@Pro` are not supported, since open-source version of JOOQ is used.
Review comment:
Yes, updated the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883144
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410888601
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
+
+JOOQ generates SQL statements based on SQL dialect determined by database connection details.
+List of supported dialects: https://www.jooq.org/javadoc/3.13.x/org.jooq/org/jooq/SQLDialect.html.
+Note: dialects annotated with `@Pro` are not supported, since open-source version of JOOQ is used.
Review comment:
Well, you can configure JDBC plugin to query these tables but I don't expect users to query Drill Metastore tables directly. They can connect to the db the regular way though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410885290
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410834619
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/operate/RdbmsRead.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.operate;
+
+import org.apache.drill.metastore.operate.AbstractRead;
+import org.apache.drill.metastore.operate.MetadataTypeValidator;
+import org.apache.drill.metastore.operate.Read;
+import org.apache.drill.metastore.rdbms.RdbmsMetastoreContext;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.apache.drill.metastore.rdbms.transform.MetadataMapper;
+import org.jooq.Condition;
+import org.jooq.DSLContext;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Result;
+import org.jooq.exception.DataAccessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Implementation of {@link Read} interface based on {@link AbstractRead} parent class.
+ * Reads information from RDBMS tables based on given filter expression.
+ * Supports reading information for specific columns.
+ *
+ * @param <T> Metastore component unit type
+ */
+public class RdbmsRead<T> extends AbstractRead<T> {
Review comment:
Very painful experience with Impala tells us that, when Drill is used under load, there must be some form of metadata cache or either a) performance suffers, or b) the DB becomes a bottleneck.
Does this design lend itself to caching? How will we know, without flushing and reloading, that metadata for some data item changed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410834268
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.SEGMENTS.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.SEGMENTS.METADATA_STATISTICS)
+ .put(MetastoreColumn.COLUMN, Tables.SEGMENTS.COLUMN)
+ .put(MetastoreColumn.LOCATIONS, Tables.SEGMENTS.LOCATIONS)
+ .put(MetastoreColumn.PARTITION_VALUES, Tables.SEGMENTS.PARTITION_VALUES)
+ .put(MetastoreColumn.PATH, Tables.SEGMENTS.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.SEGMENTS.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.SEGMENTS.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static SegmentMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<SegmentsRecord> table() {
+ return Tables.SEGMENTS;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ SegmentsRecord segmentsRecord = (SegmentsRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(segmentsRecord.getStoragePlugin())
+ .workspace(segmentsRecord.getWorkspace())
+ .tableName(segmentsRecord.getTableName())
+ .metadataKey(segmentsRecord.getMetadataKey())
+ .metadataIdentifier(segmentsRecord.getMetadataIdentifier())
+ .metadataType(segmentsRecord.getMetadataType())
+ .location(segmentsRecord.getLocation())
+ .schema(segmentsRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(segmentsRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(segmentsRecord.getMetadataStatistics()))
+ .column(segmentsRecord.getColumn())
+ .locations(ConverterUtil.convertToListString(segmentsRecord.getLocations()))
+ .partitionValues(ConverterUtil.convertToListString(segmentsRecord.getPartitionValues()))
+ .path(segmentsRecord.getPath())
+ .lastModifiedTime(segmentsRecord.getLastModifiedTime())
+ .additionalMetadata(segmentsRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public SegmentsRecord toRecord(TableMetadataUnit unit) {
+ SegmentsRecord record = new SegmentsRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataIdentifier(unit.metadataIdentifier());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setColumn(unit.column());
+ record.setLocations(ConverterUtil.convertToString(unit.locations()));
+ record.setPartitionValues(ConverterUtil.convertToString(unit.partitionValues()));
+ record.setPath(unit.path());
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return COMPONENT_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 4;
+ return Arrays.asList(Tables.SEGMENTS.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.SEGMENTS.WORKSPACE.eq(values.get(1)),
+ Tables.SEGMENTS.TABLE_NAME.eq(values.get(2)),
+ Tables.SEGMENTS.METADATA_KEY.eq(values.get(3)));
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#FILES} table.
+ */
+ public static class FileMapper extends TablesMetadataMapper<FilesRecord> {
+
+ private static final FileMapper INSTANCE = new FileMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.FILES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.FILES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.FILES.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.FILES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.FILES.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.FILES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.FILES.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.FILES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.FILES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.FILES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PATH, Tables.FILES.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.FILES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.FILES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static FileMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<FilesRecord> table() {
+ return Tables.FILES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ FilesRecord filesRecord = (FilesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(filesRecord.getStoragePlugin())
+ .workspace(filesRecord.getWorkspace())
+ .tableName(filesRecord.getTableName())
+ .metadataKey(filesRecord.getMetadataKey())
+ .metadataIdentifier(filesRecord.getMetadataIdentifier())
+ .metadataType(filesRecord.getMetadataType())
+ .location(filesRecord.getLocation())
+ .schema(filesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(filesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(filesRecord.getMetadataStatistics()))
+ .path(filesRecord.getPath())
+ .lastModifiedTime(filesRecord.getLastModifiedTime())
+ .additionalMetadata(filesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public FilesRecord toRecord(TableMetadataUnit unit) {
+ FilesRecord record = new FilesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataIdentifier(unit.metadataIdentifier());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPath(unit.path());
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return COMPONENT_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 4;
+ return Arrays.asList(
+ Tables.FILES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.FILES.WORKSPACE.eq(values.get(1)),
+ Tables.FILES.TABLE_NAME.eq(values.get(2)),
+ Tables.FILES.METADATA_KEY.eq(values.get(3)));
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#ROW_GROUPS} table.
+ */
+ public static class RowGroupMapper extends TablesMetadataMapper<RowGroupsRecord> {
+
+ private static final RowGroupMapper INSTANCE = new RowGroupMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.ROW_GROUPS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.ROW_GROUPS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.ROW_GROUPS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.ROW_GROUPS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.ROW_GROUPS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.ROW_GROUPS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.ROW_GROUPS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.ROW_GROUPS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.ROW_GROUPS.COLUMN_STATISTICS)
Review comment:
Does a row group have column statistics? And a schema?
Is this schema basically every table having the same schema? (The docs said something about this, but I thought it was for the in-memory class; didn't realize it was for the tables.)
How hard would it be to create a proper schema? Experience is screaming that generic DB tables always cause problems and precise, well-designed schemas usually turn out to be easier to maintain, evolve and use.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410832302
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
+
+JOOQ generates SQL statements based on SQL dialect determined by database connection details.
+List of supported dialects: https://www.jooq.org/javadoc/3.13.x/org.jooq/org/jooq/SQLDialect.html.
+Note: dialects annotated with `@Pro` are not supported, since open-source version of JOOQ is used.
Review comment:
Oh, and another obvious question. Can I use Drill to query the metadata tables to see what's been created? If so, how would I configure Drill to do so? Are there any issues I should know about? Is the schema subject to change without notice? Is the data joined by FK/PK relationships in the obvious way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896430
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/operate/RdbmsRead.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.operate;
+
+import org.apache.drill.metastore.operate.AbstractRead;
+import org.apache.drill.metastore.operate.MetadataTypeValidator;
+import org.apache.drill.metastore.operate.Read;
+import org.apache.drill.metastore.rdbms.RdbmsMetastoreContext;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.apache.drill.metastore.rdbms.transform.MetadataMapper;
+import org.jooq.Condition;
+import org.jooq.DSLContext;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Result;
+import org.jooq.exception.DataAccessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Implementation of {@link Read} interface based on {@link AbstractRead} parent class.
+ * Reads information from RDBMS tables based on given filter expression.
+ * Supports reading information for specific columns.
+ *
+ * @param <T> Metastore component unit type
+ */
+public class RdbmsRead<T> extends AbstractRead<T> {
Review comment:
No caching mechanism is implemented yet, necessity of it should be discussed first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410886522
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410822496
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
Review comment:
Out of the box, **the** Drill RDBMS Metastore is configured to use **the** embedded file system based SQLite database.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410829168
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
Review comment:
Drill uses the open-source version of JOOQ to generate the queries sent to the configured metastore database.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410886755
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883666
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896022
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.SEGMENTS.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.SEGMENTS.METADATA_STATISTICS)
+ .put(MetastoreColumn.COLUMN, Tables.SEGMENTS.COLUMN)
+ .put(MetastoreColumn.LOCATIONS, Tables.SEGMENTS.LOCATIONS)
+ .put(MetastoreColumn.PARTITION_VALUES, Tables.SEGMENTS.PARTITION_VALUES)
+ .put(MetastoreColumn.PATH, Tables.SEGMENTS.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.SEGMENTS.LAST_MODIFIED_TIME)
Review comment:
No. We update all data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410824015
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
Review comment:
**The** Drill Metastore ~~consists of components~~ **stores several types of metadata, called compoents**. Currently, only **the** `tables` component is implemented. ((In the RDBS implementation or overall?))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410885363
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410823604
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
Review comment:
Note: **as mentioned above, the** PostgreSQL JDBC driver must be present in **the** Drill classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410885432
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410821733
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
Review comment:
Use the connection properties to specify how Drill should connect to your Metastore database.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410826717
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
Review comment:
Liquibase ~~is using~~ **uses a** yaml ... ~~into~~ **to the** database ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410822200
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
Review comment:
Note ~~,~~ **: the** driver **class** must be included into **the** Drill classpath ~~prior to start up for all databases except of SQLite~~ . The easiest way to do that is to put the driver jar file into the `$DRILL_HOME/jars/3rdparty` folder. Or, to make upgrades easier, in your `$DRILL_SITE/jars` folder. Drill includes the driver for SQLite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410833855
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.SEGMENTS.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.SEGMENTS.METADATA_STATISTICS)
+ .put(MetastoreColumn.COLUMN, Tables.SEGMENTS.COLUMN)
+ .put(MetastoreColumn.LOCATIONS, Tables.SEGMENTS.LOCATIONS)
+ .put(MetastoreColumn.PARTITION_VALUES, Tables.SEGMENTS.PARTITION_VALUES)
+ .put(MetastoreColumn.PATH, Tables.SEGMENTS.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.SEGMENTS.LAST_MODIFIED_TIME)
Review comment:
Can a column be modified separately from its table?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410833216
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
Review comment:
If this is a table schema, it seems highly denormalized, not at all what I would expect from knowing the [HMS schema](https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png).
Seems that we want our RDBMS schema to be a proper, normalized, RDBMS-native structure, not a mapping from internal data structures to a complex schema. Though, it could be that the above entries are tables, not columns? And that something like the list of tracked ("interesting") columns is, in fact a table?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410830741
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/QueryExecutorProvider.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.jooq.DSLContext;
+import org.jooq.SQLDialect;
+import org.jooq.impl.DSL;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+
+/**
+ * Provides SQL queries executor configured based on given data source and SQL dialect.
+ */
+public class QueryExecutorProvider implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(QueryExecutorProvider.class);
+
+ private final HikariDataSource dataSource;
+ private final SQLDialect dialect;
+
+ public QueryExecutorProvider(HikariDataSource dataSource) {
+ this.dataSource = dataSource;
+ this.dialect = defineDialect();
+ }
+
+ /**
+ * Provides query executor which can be used to execute various SQL statements.
+ * Executor transforms programmatically created queries into configured SQL dialect,
+ * executes them using connections from provided data source. Allows to execute
+ * SQL queries in transaction.
+ * Note: always close executor to release open connections.
+ *
+ * @return query executor
+ */
+ public DSLContext executor() {
+ return DSL.using(dataSource, dialect);
+ }
+
+ @Override
+ public void close() {
+ dataSource.close();
+ }
+
+ /**
+ * Defines SQL dialect based on data source connection.
+ * If unable to define the dialect, uses {@link SQLDialect#DEFAULT}.
+ *
+ * @return SQL dialect
+ */
+ private SQLDialect defineDialect() {
+ SQLDialect dialect = SQLDialect.DEFAULT;
+ try (Connection connection = dataSource.getConnection()) {
+ dialect = JDBCUtils.dialect(connection);
+ } catch (SQLException e) {
+ logger.debug("Unable to connect to data source in order to define SQL dialect: {}", e.getMessage(), e);
Review comment:
This seems like exactly the kind of problem a typical user (that would be me) will have. Stuff won't work for some dumb reason, but the only indication is a debug-level log message.
Should we:
* Dedicate a logger to narrating metastore start up and errors?
* Explain in the user guide how to turn on the logger, and show what a successful startup is like.
Example:
... Found drill-metastore-override.conf in /path/to/file.
... Metastore is configured to use the MySQL database with url "...", user "bob".
... ERROR: MySQL JDBC driver not found on the class path.
Or...
... Connection to MySQL with url "..." failed. Is user "bob", no password correct?
Or...
... Database Oracle7 is not supported by Liquibase.
Or
... Unable to create Drill metastore table TABLES. Does user 'bob' have correct permissions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on issue #2060: DRILL-7706: Implement
Drill RDBMS Metastore
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #2060: DRILL-7706: Implement Drill RDBMS Metastore
URL: https://github.com/apache/drill/pull/2060#issuecomment-615394559
@arina-ielchiieva, this is great! I wonder, will this PR handle a particular use case that I'm seeing?
The metastore works for files. We are seeing more cases where users want to read other data sources: HTTP, Mongo, whatever. These sources deliver JSON, which can be ambiguous. We want to use a provided schema to resolve ambiguities. (That work is ongoing in updating the JSON reader, etc.) So, we need a place to store the provided schema.
Recall that, for files, we put the provided schema in the same folder as the data. For the HTTP plugin, say, there is no directory to place the files.
So, can we use the DB-backed metastore? Will there be a standard way for a plugin to map its concept of a table to an entry in the metastore which can hold the provided schema?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410882803
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SchemaUtilites.java
##########
@@ -218,7 +218,7 @@ private static AbstractSchema resolveToDrillSchemaInternal (SchemaPlus defaultSc
if (isRootSchema(schema)) {
throw UserException.validationError()
- .message("Root schema is immutable. Creating or dropping tables/views is not allowed in root schema." +
+ .message("Root schema is immutable. Creating or dropping tables/views is not allowed in root schema. " +
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410887241
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410828077
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
Review comment:
Since Drill will create the required tables, ensure that the database user has the following permissions in the metastore schema:
* Read and write tables
* Create and modify database objects (tables, indexes, views, etc.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410825761
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
Review comment:
In the RDBMS implementation of the Drill Metastore, the `tables` component includes five tables, one for each metadata type. The five tables are: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, **and** `PARTITIONS`. See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for the schema and indexes of each table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410828814
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
Review comment:
((Sorry, I've gotten this far, but it is not clear the actual steps I should perform. What I gather so far is that I should:
* Select a (distributed) database
* Create a DB schema (AKA database) for the Drill metastore.
* Create a user that has the required rights in that database.
* Specify metastore configuration in the indicated file.
Then the description gives great background information, but leaves me hanging: what do I do next? How do I initialize the DB? How do I verify it works? What do I check if something goes wrong (which it always does with DB-related stuff)?
Maybe move the topics around to give a list of steps, then have topics that explain the theory of operation? ))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410729508
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
Review comment:
Small assumption: table path may be specified as a table name, for example ````dfs.`/tmp/a/b/c/nation.parquet` ```` so I would recommend making its size closer to the `LOCATION` size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410887014
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410890172
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/RdbmsMetastore.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.typesafe.config.Config;
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+import liquibase.Liquibase;
+import liquibase.database.Database;
+import liquibase.database.DatabaseFactory;
+import liquibase.database.jvm.JdbcConnection;
+import liquibase.resource.ClassLoaderResourceAccessor;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.metastore.Metastore;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.views.Views;
+import org.apache.drill.metastore.config.MetastoreConfigConstants;
+import org.apache.drill.metastore.rdbms.components.tables.RdbmsTables;
+import org.apache.drill.metastore.rdbms.config.RdbmsConfigConstants;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.jooq.SQLDialect;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.sql.DataSource;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.util.Properties;
+
+/**
+ * RDBMS Drill Metastore implementation that creates necessary tables using Liquibase,
+ * initializes data source using provided config.
+ */
+public class RdbmsMetastore implements Metastore {
+
+ private static final Logger logger = LoggerFactory.getLogger(RdbmsMetastore.class);
+
+ private static final String LIQUIBASE_CHANGELOG_FILE = "db/changelog/changelog.yaml";
+
+ private final QueryExecutorProvider executorProvider;
+
+ public RdbmsMetastore(DrillConfig config) {
+ HikariDataSource dataSource = dataSource(config);
+ this.executorProvider = new QueryExecutorProvider(dataSource);
+ initTables(dataSource);
+ }
+
+ @Override
+ public Tables tables() {
+ return new RdbmsTables(executorProvider);
+ }
+
+ @Override
+ public Views views() {
+ throw new UnsupportedOperationException("Views metadata support is not implemented");
+ }
+
+ @Override
+ public void close() {
+ executorProvider.close();
+ }
+
+ /**
+ * Prepares database before initializing data source based on its type,
+ * initializes {@link HikariDataSource} instance and configures it based on given
+ * Metastore configuration.
+ * Basic parameters such as driver, url, user name and password are set using setters.
+ * Other source parameters are set dynamically through the properties. See the list
+ * of available Hikari properties: <a href="https://github.com/brettwooldridge/HikariCP">.
+ *
+ * @param config Metastore config
+ * @return Hikari data source instance
+ * @throws RdbmsMetastoreException if unable to configure Hikari data source
+ */
+ private HikariDataSource dataSource(DrillConfig config) {
+ prepareDatabase(config);
+ try {
+ Properties properties = new Properties();
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES)) {
+ Config propertiesConfig = config.getConfig(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES);
+ propertiesConfig.entrySet().forEach(e -> properties.put(e.getKey(), e.getValue().unwrapped()));
+ }
+ HikariConfig hikariConfig = new HikariConfig(properties);
+ hikariConfig.setDriverClassName(config.getString(RdbmsConfigConstants.DATA_SOURCE_DRIVER));
+ hikariConfig.setJdbcUrl(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_USER_NAME)) {
+ hikariConfig.setUsername(config.getString(RdbmsConfigConstants.DATA_SOURCE_USER_NAME));
+ }
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PASSWORD)) {
+ hikariConfig.setPassword(config.getString(RdbmsConfigConstants.DATA_SOURCE_PASSWORD));
+ }
+ return new HikariDataSource(hikariConfig);
+ } catch (RuntimeException e) {
+ throw new RdbmsMetastoreException("Unable to init RDBMS Metastore data source: " + e.getMessage(), e);
Review comment:
All Metastore implementations through their specific exceptions which extend `org.apache.drill.metastore.exceptions.MetastoreException` so than in Drill code when Metastore is called this exceptions can be caught and wrapped into the `UserException` or whatever is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410886374
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
Review comment:
In Metastore API unit is single read / write object for specific component. Table component has table unit, views - view unit, udfs - udf unit. Since table unit can define table, segment, file, row group or partition it is slightly denormalized. Metadata type defines to which type table unit belongs to. Since Metastore API is generic and should be applicable to any type of storage: file based, RDBMS based, NoSQL based such approach fits well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410700439
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
Review comment:
```suggestion
Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/changelog.yaml`.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410829652
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
+
+JOOQ generates SQL statements based on SQL dialect determined by database connection details.
+List of supported dialects: https://www.jooq.org/javadoc/3.13.x/org.jooq/org/jooq/SQLDialect.html.
+Note: dialects annotated with `@Pro` are not supported, since open-source version of JOOQ is used.
Review comment:
((This part is confusing. I'd like to know the list of supported DBs. Am I to understand that I can use any DB for which a JDBC driver is available (so I can do that part of the config), which has Liquibase support, and which has open-source JOOQ support?
Is that rather too open ended? Should we list those that Drill tests? Maybe say that any other should work if there is Liquibase and JOOQ support? Maybe add that such support is a community effort: posts questions and suggestions on the user mailing list.))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410895989
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
Review comment:
Explained above, denormalizated structure is chosen deliberately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410889818
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/QueryExecutorProvider.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.jooq.DSLContext;
+import org.jooq.SQLDialect;
+import org.jooq.impl.DSL;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+
+/**
+ * Provides SQL queries executor configured based on given data source and SQL dialect.
+ */
+public class QueryExecutorProvider implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(QueryExecutorProvider.class);
+
+ private final HikariDataSource dataSource;
+ private final SQLDialect dialect;
+
+ public QueryExecutorProvider(HikariDataSource dataSource) {
+ this.dataSource = dataSource;
+ this.dialect = defineDialect();
+ }
+
+ /**
+ * Provides query executor which can be used to execute various SQL statements.
+ * Executor transforms programmatically created queries into configured SQL dialect,
+ * executes them using connections from provided data source. Allows to execute
+ * SQL queries in transaction.
+ * Note: always close executor to release open connections.
+ *
+ * @return query executor
+ */
+ public DSLContext executor() {
+ return DSL.using(dataSource, dialect);
+ }
+
+ @Override
+ public void close() {
+ dataSource.close();
+ }
+
+ /**
+ * Defines SQL dialect based on data source connection.
+ * If unable to define the dialect, uses {@link SQLDialect#DEFAULT}.
+ *
+ * @return SQL dialect
+ */
+ private SQLDialect defineDialect() {
+ SQLDialect dialect = SQLDialect.DEFAULT;
+ try (Connection connection = dataSource.getConnection()) {
+ dialect = JDBCUtils.dialect(connection);
+ } catch (SQLException e) {
+ logger.debug("Unable to connect to data source in order to define SQL dialect: {}", e.getMessage(), e);
Review comment:
Log already contains information about which confess were used (`DrillConfig#create` is responsible for this). Also which SQL dialect was used. Other configuration is not logged for security reasons.
Regarding better error handling, I think this can be done as future improvement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410824795
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
Review comment:
((Sorry, didn't understand this. Does this mean that `TableMetadataUnit` implements the tables component? Describes it? What does "Unit" mean? And, is the **Table** metadata unit applicable for any metadata type in the sense of those others that are not implemented yet? Or, is this the class that holds fields for all of the objects within the table component, meaning it has the union of fields for tables, segments, files, row groups and partitions? If so, I might have a suggestion about an improved design...))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410831008
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/RdbmsMetastore.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.typesafe.config.Config;
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+import liquibase.Liquibase;
+import liquibase.database.Database;
+import liquibase.database.DatabaseFactory;
+import liquibase.database.jvm.JdbcConnection;
+import liquibase.resource.ClassLoaderResourceAccessor;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.metastore.Metastore;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.views.Views;
+import org.apache.drill.metastore.config.MetastoreConfigConstants;
+import org.apache.drill.metastore.rdbms.components.tables.RdbmsTables;
+import org.apache.drill.metastore.rdbms.config.RdbmsConfigConstants;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.jooq.SQLDialect;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.sql.DataSource;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.util.Properties;
+
+/**
+ * RDBMS Drill Metastore implementation that creates necessary tables using Liquibase,
+ * initializes data source using provided config.
+ */
+public class RdbmsMetastore implements Metastore {
+
+ private static final Logger logger = LoggerFactory.getLogger(RdbmsMetastore.class);
+
+ private static final String LIQUIBASE_CHANGELOG_FILE = "db/changelog/changelog.yaml";
+
+ private final QueryExecutorProvider executorProvider;
+
+ public RdbmsMetastore(DrillConfig config) {
+ HikariDataSource dataSource = dataSource(config);
+ this.executorProvider = new QueryExecutorProvider(dataSource);
+ initTables(dataSource);
+ }
+
+ @Override
+ public Tables tables() {
+ return new RdbmsTables(executorProvider);
+ }
+
+ @Override
+ public Views views() {
+ throw new UnsupportedOperationException("Views metadata support is not implemented");
+ }
+
+ @Override
+ public void close() {
+ executorProvider.close();
+ }
+
+ /**
+ * Prepares database before initializing data source based on its type,
+ * initializes {@link HikariDataSource} instance and configures it based on given
+ * Metastore configuration.
+ * Basic parameters such as driver, url, user name and password are set using setters.
+ * Other source parameters are set dynamically through the properties. See the list
+ * of available Hikari properties: <a href="https://github.com/brettwooldridge/HikariCP">.
+ *
+ * @param config Metastore config
+ * @return Hikari data source instance
+ * @throws RdbmsMetastoreException if unable to configure Hikari data source
+ */
+ private HikariDataSource dataSource(DrillConfig config) {
+ prepareDatabase(config);
+ try {
+ Properties properties = new Properties();
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES)) {
+ Config propertiesConfig = config.getConfig(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES);
+ propertiesConfig.entrySet().forEach(e -> properties.put(e.getKey(), e.getValue().unwrapped()));
+ }
+ HikariConfig hikariConfig = new HikariConfig(properties);
+ hikariConfig.setDriverClassName(config.getString(RdbmsConfigConstants.DATA_SOURCE_DRIVER));
+ hikariConfig.setJdbcUrl(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_USER_NAME)) {
+ hikariConfig.setUsername(config.getString(RdbmsConfigConstants.DATA_SOURCE_USER_NAME));
+ }
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PASSWORD)) {
+ hikariConfig.setPassword(config.getString(RdbmsConfigConstants.DATA_SOURCE_PASSWORD));
+ }
+ return new HikariDataSource(hikariConfig);
+ } catch (RuntimeException e) {
+ throw new RdbmsMetastoreException("Unable to init RDBMS Metastore data source: " + e.getMessage(), e);
Review comment:
This is exactly the kind of information we'll need when a user on the mailing list runs into problems. But, we're not providing much context. Consider using `UserException` so that we can add lots of context and to ensure the error is logged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410883620
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896227
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static TableMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<TablesRecord> table() {
+ return Tables.TABLES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ TablesRecord tablesRecord = (TablesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(tablesRecord.getStoragePlugin())
+ .workspace(tablesRecord.getWorkspace())
+ .tableName(tablesRecord.getTableName())
+ .owner(tablesRecord.getOwner())
+ .tableType(tablesRecord.getTableType())
+ .metadataKey(tablesRecord.getMetadataKey())
+ .metadataType(tablesRecord.getMetadataType())
+ .location(tablesRecord.getLocation())
+ .interestingColumns(ConverterUtil.convertToListString(tablesRecord.getInterestingColumns()))
+ .schema(tablesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(tablesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(tablesRecord.getMetadataStatistics()))
+ .partitionKeys(ConverterUtil.convertToMapStringString(tablesRecord.getPartitionKeys()))
+ .lastModifiedTime(tablesRecord.getLastModifiedTime())
+ .additionalMetadata(tablesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public TablesRecord toRecord(TableMetadataUnit unit) {
+ TablesRecord record = new TablesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setOwner(unit.owner());
+ record.setTableType(unit.tableType());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setInterestingColumns(ConverterUtil.convertToString(unit.interestingColumns()));
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPartitionKeys(ConverterUtil.convertToString(unit.partitionKeys()));
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return TABLE_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 3;
+ return Arrays.asList(
+ Tables.TABLES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.TABLES.WORKSPACE.eq(values.get(1)),
+ Tables.TABLES.TABLE_NAME.eq(values.get(2))
+ );
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#SEGMENTS} table.
+ */
+ public static class SegmentMapper extends TablesMetadataMapper<SegmentsRecord> {
+
+ private static final SegmentMapper INSTANCE = new SegmentMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.SEGMENTS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.SEGMENTS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.SEGMENTS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.SEGMENTS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.SEGMENTS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.SEGMENTS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.SEGMENTS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.SEGMENTS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.SEGMENTS.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.SEGMENTS.METADATA_STATISTICS)
+ .put(MetastoreColumn.COLUMN, Tables.SEGMENTS.COLUMN)
+ .put(MetastoreColumn.LOCATIONS, Tables.SEGMENTS.LOCATIONS)
+ .put(MetastoreColumn.PARTITION_VALUES, Tables.SEGMENTS.PARTITION_VALUES)
+ .put(MetastoreColumn.PATH, Tables.SEGMENTS.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.SEGMENTS.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.SEGMENTS.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static SegmentMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<SegmentsRecord> table() {
+ return Tables.SEGMENTS;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ SegmentsRecord segmentsRecord = (SegmentsRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(segmentsRecord.getStoragePlugin())
+ .workspace(segmentsRecord.getWorkspace())
+ .tableName(segmentsRecord.getTableName())
+ .metadataKey(segmentsRecord.getMetadataKey())
+ .metadataIdentifier(segmentsRecord.getMetadataIdentifier())
+ .metadataType(segmentsRecord.getMetadataType())
+ .location(segmentsRecord.getLocation())
+ .schema(segmentsRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(segmentsRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(segmentsRecord.getMetadataStatistics()))
+ .column(segmentsRecord.getColumn())
+ .locations(ConverterUtil.convertToListString(segmentsRecord.getLocations()))
+ .partitionValues(ConverterUtil.convertToListString(segmentsRecord.getPartitionValues()))
+ .path(segmentsRecord.getPath())
+ .lastModifiedTime(segmentsRecord.getLastModifiedTime())
+ .additionalMetadata(segmentsRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public SegmentsRecord toRecord(TableMetadataUnit unit) {
+ SegmentsRecord record = new SegmentsRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataIdentifier(unit.metadataIdentifier());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setColumn(unit.column());
+ record.setLocations(ConverterUtil.convertToString(unit.locations()));
+ record.setPartitionValues(ConverterUtil.convertToString(unit.partitionValues()));
+ record.setPath(unit.path());
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return COMPONENT_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 4;
+ return Arrays.asList(Tables.SEGMENTS.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.SEGMENTS.WORKSPACE.eq(values.get(1)),
+ Tables.SEGMENTS.TABLE_NAME.eq(values.get(2)),
+ Tables.SEGMENTS.METADATA_KEY.eq(values.get(3)));
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#FILES} table.
+ */
+ public static class FileMapper extends TablesMetadataMapper<FilesRecord> {
+
+ private static final FileMapper INSTANCE = new FileMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.FILES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.FILES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.FILES.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.FILES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.FILES.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.FILES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.FILES.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.FILES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.FILES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.FILES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PATH, Tables.FILES.PATH)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.FILES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.FILES.ADDITIONAL_METADATA)
+ .build();
+
+ private static final RdbmsFilterExpressionVisitor FILTER_VISITOR = new RdbmsFilterExpressionVisitor(COLUMNS_MAP);
+
+ public static FileMapper get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Table<FilesRecord> table() {
+ return Tables.FILES;
+ }
+
+ @Override
+ public TableMetadataUnit toUnit(Record record) {
+ FilesRecord filesRecord = (FilesRecord) record;
+ return TableMetadataUnit.builder()
+ .storagePlugin(filesRecord.getStoragePlugin())
+ .workspace(filesRecord.getWorkspace())
+ .tableName(filesRecord.getTableName())
+ .metadataKey(filesRecord.getMetadataKey())
+ .metadataIdentifier(filesRecord.getMetadataIdentifier())
+ .metadataType(filesRecord.getMetadataType())
+ .location(filesRecord.getLocation())
+ .schema(filesRecord.getSchema())
+ .columnsStatistics(ConverterUtil.convertToMapStringString(filesRecord.getColumnStatistics()))
+ .metadataStatistics(ConverterUtil.convertToListString(filesRecord.getMetadataStatistics()))
+ .path(filesRecord.getPath())
+ .lastModifiedTime(filesRecord.getLastModifiedTime())
+ .additionalMetadata(filesRecord.getAdditionalMetadata())
+ .build();
+ }
+
+ @Override
+ public FilesRecord toRecord(TableMetadataUnit unit) {
+ FilesRecord record = new FilesRecord();
+ record.setStoragePlugin(unit.storagePlugin());
+ record.setWorkspace(unit.workspace());
+ record.setTableName(unit.tableName());
+ record.setMetadataKey(unit.metadataKey());
+ record.setMetadataIdentifier(unit.metadataIdentifier());
+ record.setMetadataType(unit.metadataType());
+ record.setLocation(unit.location());
+ record.setSchema(unit.schema());
+ record.setColumnStatistics(ConverterUtil.convertToString(unit.columnsStatistics()));
+ record.setMetadataStatistics(ConverterUtil.convertToString(unit.metadataStatistics()));
+ record.setPath(unit.path());
+ record.setLastModifiedTime(unit.lastModifiedTime());
+ record.setAdditionalMetadata(unit.additionalMetadata());
+ return record;
+ }
+
+ @Override
+ protected Map<MetastoreColumn, Field<?>> fieldMapper() {
+ return COLUMNS_MAP;
+ }
+
+ @Override
+ protected RdbmsFilterExpressionVisitor filterVisitor() {
+ return FILTER_VISITOR;
+ }
+
+ @Override
+ protected Function<TableMetadataUnit, List<String>> partitionKey() {
+ return COMPONENT_PARTITION_KEY;
+ }
+
+ @Override
+ protected List<Condition> toConditions(List<String> values) {
+ assert values.size() == 4;
+ return Arrays.asList(
+ Tables.FILES.STORAGE_PLUGIN.eq(values.get(0)),
+ Tables.FILES.WORKSPACE.eq(values.get(1)),
+ Tables.FILES.TABLE_NAME.eq(values.get(2)),
+ Tables.FILES.METADATA_KEY.eq(values.get(3)));
+ }
+ }
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#ROW_GROUPS} table.
+ */
+ public static class RowGroupMapper extends TablesMetadataMapper<RowGroupsRecord> {
+
+ private static final RowGroupMapper INSTANCE = new RowGroupMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.ROW_GROUPS.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.ROW_GROUPS.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.ROW_GROUPS.TABLE_NAME)
+ .put(MetastoreColumn.METADATA_KEY, Tables.ROW_GROUPS.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_IDENTIFIER, Tables.ROW_GROUPS.METADATA_IDENTIFIER)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.ROW_GROUPS.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.ROW_GROUPS.LOCATION)
+ .put(MetastoreColumn.SCHEMA, Tables.ROW_GROUPS.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.ROW_GROUPS.COLUMN_STATISTICS)
Review comment:
Yes. Please see `org.apache.drill.metastore.metadata.RowGroupMetadata` class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410896681
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410890227
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/RdbmsMetastore.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms;
+
+import com.typesafe.config.Config;
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+import liquibase.Liquibase;
+import liquibase.database.Database;
+import liquibase.database.DatabaseFactory;
+import liquibase.database.jvm.JdbcConnection;
+import liquibase.resource.ClassLoaderResourceAccessor;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.metastore.Metastore;
+import org.apache.drill.metastore.components.tables.Tables;
+import org.apache.drill.metastore.components.views.Views;
+import org.apache.drill.metastore.config.MetastoreConfigConstants;
+import org.apache.drill.metastore.rdbms.components.tables.RdbmsTables;
+import org.apache.drill.metastore.rdbms.config.RdbmsConfigConstants;
+import org.apache.drill.metastore.rdbms.exception.RdbmsMetastoreException;
+import org.jooq.SQLDialect;
+import org.jooq.tools.jdbc.JDBCUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.sql.DataSource;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.util.Properties;
+
+/**
+ * RDBMS Drill Metastore implementation that creates necessary tables using Liquibase,
+ * initializes data source using provided config.
+ */
+public class RdbmsMetastore implements Metastore {
+
+ private static final Logger logger = LoggerFactory.getLogger(RdbmsMetastore.class);
+
+ private static final String LIQUIBASE_CHANGELOG_FILE = "db/changelog/changelog.yaml";
+
+ private final QueryExecutorProvider executorProvider;
+
+ public RdbmsMetastore(DrillConfig config) {
+ HikariDataSource dataSource = dataSource(config);
+ this.executorProvider = new QueryExecutorProvider(dataSource);
+ initTables(dataSource);
+ }
+
+ @Override
+ public Tables tables() {
+ return new RdbmsTables(executorProvider);
+ }
+
+ @Override
+ public Views views() {
+ throw new UnsupportedOperationException("Views metadata support is not implemented");
+ }
+
+ @Override
+ public void close() {
+ executorProvider.close();
+ }
+
+ /**
+ * Prepares database before initializing data source based on its type,
+ * initializes {@link HikariDataSource} instance and configures it based on given
+ * Metastore configuration.
+ * Basic parameters such as driver, url, user name and password are set using setters.
+ * Other source parameters are set dynamically through the properties. See the list
+ * of available Hikari properties: <a href="https://github.com/brettwooldridge/HikariCP">.
+ *
+ * @param config Metastore config
+ * @return Hikari data source instance
+ * @throws RdbmsMetastoreException if unable to configure Hikari data source
+ */
+ private HikariDataSource dataSource(DrillConfig config) {
+ prepareDatabase(config);
+ try {
+ Properties properties = new Properties();
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES)) {
+ Config propertiesConfig = config.getConfig(RdbmsConfigConstants.DATA_SOURCE_PROPERTIES);
+ propertiesConfig.entrySet().forEach(e -> properties.put(e.getKey(), e.getValue().unwrapped()));
+ }
+ HikariConfig hikariConfig = new HikariConfig(properties);
+ hikariConfig.setDriverClassName(config.getString(RdbmsConfigConstants.DATA_SOURCE_DRIVER));
+ hikariConfig.setJdbcUrl(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_USER_NAME)) {
+ hikariConfig.setUsername(config.getString(RdbmsConfigConstants.DATA_SOURCE_USER_NAME));
+ }
+ if (config.hasPath(RdbmsConfigConstants.DATA_SOURCE_PASSWORD)) {
+ hikariConfig.setPassword(config.getString(RdbmsConfigConstants.DATA_SOURCE_PASSWORD));
+ }
+ return new HikariDataSource(hikariConfig);
+ } catch (RuntimeException e) {
+ throw new RdbmsMetastoreException("Unable to init RDBMS Metastore data source: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Prepares database prior to data source configuration based on its type.
+ * For example, SQLite database requires database path to exists prior to database initialization.
+ *
+ * @param config Metastore config
+ */
+ private void prepareDatabase(DrillConfig config) {
+ SQLDialect dialect = JDBCUtils.dialect(config.getString(RdbmsConfigConstants.DATA_SOURCE_URL));
+ switch (dialect) {
+ case SQLITE:
Review comment:
Agree. Updated the code as you suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410825366
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
Review comment:
((This does indicate that it is a union of all fields. If so, then:)) The `TableMetadataUnit` class holds fields for all five metadata types within the `tables` component. Any fields not applicable to a particular metadata type are simply ignored and remain unset.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410888054
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
Review comment:
This readme is mostly intended for the developers to explain how RDBMS Metastore is implemented not for the uses on how to work with Drill Metastore. But yes, your description is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410887464
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410895858
##########
File path: metastore/rdbms-metastore/src/main/java/org/apache/drill/metastore/rdbms/components/tables/TablesMetadataMapper.java
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.metastore.rdbms.components.tables;
+
+import org.apache.drill.metastore.MetastoreColumn;
+import org.apache.drill.metastore.components.tables.TableMetadataUnit;
+import org.apache.drill.metastore.rdbms.transform.AbstractMetadataMapper;
+import org.apache.drill.metastore.rdbms.transform.RdbmsFilterExpressionVisitor;
+import org.apache.drill.metastore.rdbms.util.ConverterUtil;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.jooq.Condition;
+import org.jooq.Field;
+import org.jooq.Record;
+import org.jooq.Table;
+import org.jooq.generated.Tables;
+import org.jooq.generated.tables.records.FilesRecord;
+import org.jooq.generated.tables.records.PartitionsRecord;
+import org.jooq.generated.tables.records.RowGroupsRecord;
+import org.jooq.generated.tables.records.SegmentsRecord;
+import org.jooq.generated.tables.records.TablesRecord;
+import org.jooq.impl.DSL;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Abstract implementation of {@link AbstractMetadataMapper} for RDBMS Metastore tables component.
+ * Contains common code for specific RDBMS Metastore tables component tables.
+ *
+ * @param <R> RDBMS table record type
+ */
+public abstract class TablesMetadataMapper<R extends Record> extends AbstractMetadataMapper<TableMetadataUnit, R> {
+
+ protected static final Function<TableMetadataUnit, List<String>> TABLE_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName());
+
+ protected static final Function<TableMetadataUnit, List<String>> COMPONENT_PARTITION_KEY = unit ->
+ Arrays.asList(unit.storagePlugin(), unit.workspace(), unit.tableName(), unit.metadataKey());
+
+ @Override
+ public TableMetadataUnit emptyUnit() {
+ return TableMetadataUnit.EMPTY_UNIT;
+ }
+
+ @Override
+ public List<Condition> toDeleteConditions(List<TableMetadataUnit> units) {
+ Set<List<String>> partitionValues = units.stream()
+ .collect(Collectors.groupingBy(partitionKey(), Collectors.toList()))
+ .keySet();
+
+ return partitionValues.stream()
+ .map(values -> DSL.and(toConditions(values)))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return function to determine partition key for specific table
+ */
+ protected abstract Function<TableMetadataUnit, List<String>> partitionKey();
+
+ /**
+ * Creates JOOQ conditions based on given list of partition values.
+ * Matching is order based.
+ *
+ * @param values partition values
+ * @return list of JOOQ conditions
+ */
+ protected abstract List<Condition> toConditions(List<String> values);
+
+ /**
+ * {@link TablesMetadataMapper} implementation for {@link Tables#TABLES} table.
+ */
+ public static class TableMapper extends TablesMetadataMapper<TablesRecord> {
+
+ private static final TableMapper INSTANCE = new TableMapper();
+
+ private static final Map<MetastoreColumn, Field<?>> COLUMNS_MAP = ImmutableMap.<MetastoreColumn, Field<?>>builder()
+ .put(MetastoreColumn.STORAGE_PLUGIN, Tables.TABLES.STORAGE_PLUGIN)
+ .put(MetastoreColumn.WORKSPACE, Tables.TABLES.WORKSPACE)
+ .put(MetastoreColumn.TABLE_NAME, Tables.TABLES.TABLE_NAME)
+ .put(MetastoreColumn.OWNER, Tables.TABLES.OWNER)
+ .put(MetastoreColumn.TABLE_TYPE, Tables.TABLES.TABLE_TYPE)
+ .put(MetastoreColumn.METADATA_KEY, Tables.TABLES.METADATA_KEY)
+ .put(MetastoreColumn.METADATA_TYPE, Tables.TABLES.METADATA_TYPE)
+ .put(MetastoreColumn.LOCATION, Tables.TABLES.LOCATION)
+ .put(MetastoreColumn.INTERESTING_COLUMNS, Tables.TABLES.INTERESTING_COLUMNS)
+ .put(MetastoreColumn.SCHEMA, Tables.TABLES.SCHEMA)
+ .put(MetastoreColumn.COLUMNS_STATISTICS, Tables.TABLES.COLUMN_STATISTICS)
+ .put(MetastoreColumn.METADATA_STATISTICS, Tables.TABLES.METADATA_STATISTICS)
+ .put(MetastoreColumn.PARTITION_KEYS, Tables.TABLES.PARTITION_KEYS)
+ .put(MetastoreColumn.LAST_MODIFIED_TIME, Tables.TABLES.LAST_MODIFIED_TIME)
+ .put(MetastoreColumn.ADDITIONAL_METADATA, Tables.TABLES.ADDITIONAL_METADATA)
+ .build();
Review comment:
Yes, our RDBMS Metastore is more like blob storage for large fields and regular storage for fields by which we do filter the data. Denormailization here was chosen deliberately taking into account semantics of the Metastore API implementation and the way data is stored / deleted and accessed. Having normalized structure will be an overhead since we mostly extract all data for table, the same way when we update data - we don't do update specific fields, we just erase all data and re-write.
If we used normalized structure we would end up as HMS Metastore with many tables and relations but since we always need all data, we would have to do a lot of joins all the time. Since many queries to Metastore are generated at runtime, it would be a challenge to find framework that would handle this nicely, maybe Hibernate but still it won't cover all cases.
The same would apply for data update or deletion. First delete data from all tables then insert, update PR / FK relations. This would impact performance significantly.
It's worth mentioning that Metastore API is generic to any type of storage: file based, RDBMS based or NoSQL based. This way sometimes to make sure all works we should sacrifice something, in our case normalization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410835080
##########
File path: metastore/rdbms-metastore/src/main/resources/db/changelog/changes/initial_ddls.yaml
##########
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Creates 5 RDBMS Metastore tables component tables: TABLES, SEGMENTS, FILES, ROW_GROUPS, PARTITIONS.
+databaseChangeLog:
+ # to preserve upper case naming in some DBs (ex: PostgreSQL) and quote reserved column names (ex: COLUMN)
+ - objectQuotingStrategy: QUOTE_ALL_OBJECTS
+ - changeSet:
+ id: 1
+ author: arina
+ changes:
+ - createTable:
+ tableName: TABLES
+ columns:
+ - column:
+ name: STORAGE_PLUGIN
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: WORKSPACE
+ type: VARCHAR(100)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: TABLE_NAME
+ type: VARCHAR(200)
+ constraints:
+ primaryKey: true
+ nullable: false
+ - column:
+ name: OWNER
+ type: VARCHAR(100)
+ - column:
+ name: TABLE_TYPE
+ type: VARCHAR(100)
+ - column:
+ name: METADATA_KEY
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: METADATA_TYPE
+ type: VARCHAR(100)
+ constraints:
+ nullable: false
+ - column:
+ name: LOCATION
+ type: VARCHAR(500)
+ - column:
+ name: INTERESTING_COLUMNS
+ type: CLOB
Review comment:
As commented on earlier, this is a sub-optimal design decision.
The column sizes give above are also quite large. I suspect we may find that this schema won't perform well and will be costly to update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410821147
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
Review comment:
**The** RDBMS Metastore implementation allows **you** store Drill Metastore metadata in **a** configured RDBMS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2060:
DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410931209
##########
File path: pom.xml
##########
@@ -605,7 +605,7 @@
<configuration>
<rules>
<bannedDependencies>
- <excludes>
+ <excludes>SchemaUtilites.java
Review comment:
Looks like it was changed accidentally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2060: DRILL-7706: Implement Drill RDBMS Metastore for Tables component
URL: https://github.com/apache/drill/pull/2060#discussion_r410888270
##########
File path: metastore/rdbms-metastore/README.md
##########
@@ -0,0 +1,140 @@
+# RDBMS Metastore
+
+RDBMS Metastore implementation allows to store Drill Metastore metadata in configured RDBMS.
+
+## Configuration
+
+Currently, RDBMS Metastore is not default Drill Metastore implementation.
+To enable RDBMS Metastore create `drill-metastore-override.conf` and indicate RDBMS Metastore class:
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore"
+}
+```
+
+### Connection properties
+
+Data source connection properties allows to indicate how to connect to Drill Metastore database.
+
+`drill.metastore.rdbms.data_source.driver` - driver class name. Required.
+Note, driver must be included into Drill classpath prior to start up for all databases except of SQLite.
+
+`drill.metastore.rdbms.data_source.url` - connection url. Required.
+
+`drill.metastore.rdbms.data_source.username` - database user on whose behalf the connection is
+being made. Optional, if database does not require user to connect.
+
+`drill.metastore.rdbms.data_source.password` - database user's password.
+Optional, if database does not require user's password to connect.
+
+`drill.metastore.rdbms.data_source.properties` - specifies properties which will be used
+during data source creation. See list of available [Hikari properties](https://github.com/brettwooldridge/HikariCP)
+for more details.
+
+### Default configuration
+
+Out of the box, Drill RDBMS Metastore is configured to use embedded file system based SQLite database.
+It will be created locally in user's home directory under `${drill.exec.zk.root}"/metastore` location.
+
+Default setup can be used only in Drill embedded mode.
+If SQLite setup will be used in distributed mode, each drillbit will have it's own SQLite instance
+which will lead to bogus results during queries execution.
+In distributed mode, database instance must be accessible for all drillbits.
+
+### Custom configuration
+
+`drill-metastore-override.conf` is used to customize connection details to the Drill Metastore database.
+See `drill-metastore-override-example.conf` for more details.
+
+#### Example of PostgreSQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "org.postgresql.Driver",
+ url: "jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: PostgreSQL JDBC driver must be present in Drill classpath.
+
+#### Example of MySQL configuration
+
+```
+drill.metastore: {
+ implementation.class: "org.apache.drill.metastore.rdbms.RdbmsMetastore",
+ rdbms: {
+ data_source: {
+ driver: "com.mysql.cj.jdbc.Driver",
+ url: "jdbc:mysql://localhost:1234/drill_metastore",
+ username: "user",
+ password: "password"
+ }
+ }
+}
+```
+
+Note: MySQL JDBC driver must be present in Drill classpath.
+
+##### Driver version
+
+For MySQL connector version 6+, use `com.mysql.cj.jdbc.Driver` driver class,
+for older versions use `com.mysql.jdbc.Driver`.
+
+## Tables structure
+
+Drill Metastore consists of components. Currently, only `tables` component is implemented.
+This component provides metadata about Drill tables, including their segments, files, row groups and partitions.
+In Drill `tables` component unit is represented by `TableMetadataUnit` class which is applicable to any metadata type.
+Fields which are not applicable to particular metadata type, remain unset.
+
+In RDBMS Drill Metastore each `tables` component metadata type has it's own table.
+There are five tables: `TABLES`, `SEGMENTS`, `FILES`, `ROW_GROUPS`, `PARTITIONS`.
+These tables structure and primary keys are defined based on fields specific for each metadata type.
+See `src/main/resources/db/changelog/changes/initial_ddls.yaml` for more details.
+
+### Tables creation
+
+RDBMS Metastore is using [Liquibase](https://www.liquibase.org/documentation/core-concepts/index.html)
+to create all needed tables during RDBMS Metastore initialization, users should not create any tables manually.
+
+### Database schema
+
+Liquibase is using yaml configuration file to apply changes into database schema: `src/main/resources/db/changelog/chnagelog.yaml`.
+It converts yaml content into DDL / DML commands suitable to database syntax based on connection details.
+See list of supported databases: https://www.liquibase.org/databases.html.
+
+All Drill Metastore tables will be created in database default schema, unless schema is set in the connection url.
+It's recommended prior to using RDBMS Metastore to create dedicated Drill Metastore schema, for example, `drill_metastore`
+and indicate it in the connection url.
+
+Example:
+
+PostgreSQL: `jdbc:postgresql://localhost:1234/mydb?currentSchema=drill_metastore`
+
+MySQL: `jdbc:mysql://localhost:1234/drill_metastore`
+
+Note: database user must have not only read / write permission in Drill Metastore schema
+but also permission to create / modify database objects.
+
+### Liquibase tables
+
+During Drill RDBMS Metastore initialization, Liquibase will create two internal tracking tables:
+`DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. They are needed to track schema changes and concurrent updates.
+See https://www.liquibase.org/get_started/how-lb-works.html for more details.
+
+## Query execution
+
+SQL queries issued to RDBMS Metastore tables are generated using [JOOQ](https://www.jooq.org/doc/3.13/manual/getting-started/).
+JOOQ allows programmatically construct SQL queries at runtime and provides executor to run generated queries.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services