You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/26 21:16:37 UTC
[GitHub] [arrow] alamb opened a new pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables
alamb opened a new pull request #9818:
URL: https://github.com/apache/arrow/pull/9818
# Rationale
Provide configurable access to a table list so a user can see what tables exist).
See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#
I plan to add support for `SHOW TABLES` (and `information_schema.columns` / `SHOW COLUMNS`) as follow on PRs
# Example use:
Setup:
```
echo "1" > /tmp/table.csv
echo "2" >> /tmp/table.csv
cargo run --bin datafusion-cli
```
Then run :
```
> CREATE EXTERNAL TABLE t(a int)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select * from information_schema.tables;
+---------------+--------------------+------------+--------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+------------+--------------+
| datafusion | public | t | BASE TABLE |
| datafusion | information_schema | TABLES | SYSTEM TABLE |
+---------------+--------------------+------------+--------------+
2 row in set. Query took 0 seconds.
z>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-809433172
@jorgecarleitao -- @returnString did a great job explaining the rationale for schemas / catalogs from my perspective.
Thanks for checking in on us and making sure it is clear what is going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-810157879
Integration test failure seems unrelated
https://github.com/apache/arrow/pull/9818/checks?check_run_id=2219436309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-808513138
https://issues.apache.org/jira/browse/ARROW-12106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9818:
URL: https://github.com/apache/arrow/pull/9818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602904974
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
Review comment:
The string builder requires a number here but the number or rows isn't known. I'll add a comment explaining 10 is some arbitrary number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-808513407
FYI @seddonm1 @returnString @Dandandan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602612055
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -23,6 +23,67 @@ use std::any::Any;
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
+/// Represent a list of named catalogs
+pub trait CatalogList: Sync + Send {
+ /// Returns the catalog list as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Adds a new catalog to this catalog list
+ /// If a catalog of the same name existed before, it is replaced in the list and returned.
+ fn register_catalog(
+ &self,
+ name: String,
+ catalog: Arc<dyn CatalogProvider>,
+ ) -> Option<Arc<dyn CatalogProvider>>;
+
+ /// Retrieves the list of available catalog names
+ fn catalog_names(&self) -> Vec<String>;
Review comment:
Better as `&[&str]` to avoid clone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] seddonm1 commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-808522553
@alamb my read through looked good (as expected). I can have a play with it early next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602905367
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
Review comment:
Why not use `0` to keep it simple?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602612385
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -23,6 +23,67 @@ use std::any::Any;
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
+/// Represent a list of named catalogs
+pub trait CatalogList: Sync + Send {
+ /// Returns the catalog list as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Adds a new catalog to this catalog list
+ /// If a catalog of the same name existed before, it is replaced in the list and returned.
+ fn register_catalog(
+ &self,
+ name: String,
+ catalog: Arc<dyn CatalogProvider>,
+ ) -> Option<Arc<dyn CatalogProvider>>;
+
+ /// Retrieves the list of available catalog names
+ fn catalog_names(&self) -> Vec<String>;
Review comment:
ah I see in the implementation it is not really possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602903856
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
I think `LOCAL TEMPORARY` is referring to what happens when you do `CREATE TEMPORARY TABLE...` in postgres -- so I agree with @returnString that LOCAL TEMPORARY is probably not what we would want.
I will change the information table to say `VIEW` for the information_schema tables, consistent with Postgres (and SQL server)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r603203922
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -678,20 +695,31 @@ impl ExecutionContextState {
.resolve(&self.config.default_catalog, &self.config.default_schema)
}
+ fn catalog_for_ref<'a>(
+ &'a self,
+ resolved_ref: &ResolvedTableReference<'a>,
+ ) -> Result<Arc<dyn CatalogProvider>> {
+ if self.config.information_schema {
+ CatalogListWithInformationSchema::new(self.catalog_list.clone())
Review comment:
in https://github.com/apache/arrow/pull/9818/commits/5151f31eecfbcd0d5892837f951bb0e5ea8473c0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602904783
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -678,20 +695,31 @@ impl ExecutionContextState {
.resolve(&self.config.default_catalog, &self.config.default_schema)
}
+ fn catalog_for_ref<'a>(
+ &'a self,
+ resolved_ref: &ResolvedTableReference<'a>,
+ ) -> Result<Arc<dyn CatalogProvider>> {
+ if self.config.information_schema {
+ CatalogListWithInformationSchema::new(self.catalog_list.clone())
Review comment:
Since the `Config` can't be changed after the `ExecutionContext` is created I think this would be a good idea. I will give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] seddonm1 commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602598700
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
I have not really spent much time with the information_schema. Are the own two types `BASE TABLE` and `SYSTEM TABLE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] seddonm1 commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602616110
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
@alamb Here is a link to the Postgres doc: https://www.postgresql.org/docs/current/infoschema-tables.html
Type of the table:
- `BASE TABLE` for a persistent base table (the normal table type)
- `VIEW` for a view
- `FOREIGN` for a foreign table
- `LOCAL TEMPORARY` for a temporary table
Are all our 'tables' kind of `LOCAL TEMPORARY`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602600941
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
I am not sure to be honest. Postgres has `BASE TABLE` and `VIEW` that I saw.
I will spend some time figuring out how what the values of these should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-808926077
> I could definitely see an argument that the csv/parquet TableProviders are temporary, but arbitrary TableProviders might use totally different semantics (e.g. in my case, fetching data from a persistent store that could be classed as a base table). If we want to support different options here, perhaps we need a new TableProvider method to indicate that?
Yes, I agree -- I see no reason to support different values for `TABLE_TYPE` in the initial PR -- and if someone wants that functionality they can perhaps propose a PR to add a methods to the `TableProvider`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602904112
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -23,6 +23,67 @@ use std::any::Any;
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
+/// Represent a list of named catalogs
+pub trait CatalogList: Sync + Send {
+ /// Returns the catalog list as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Adds a new catalog to this catalog list
+ /// If a catalog of the same name existed before, it is replaced in the list and returned.
+ fn register_catalog(
+ &self,
+ name: String,
+ catalog: Arc<dyn CatalogProvider>,
+ ) -> Option<Arc<dyn CatalogProvider>>;
+
+ /// Retrieves the list of available catalog names
+ fn catalog_names(&self) -> Vec<String>;
Review comment:
The way I could think of to avoid the copy is don't make a `CatalogList` trait and instead pass around `MemoryCatalogList` directly. This would allow a function like `fn catalog_names(&self) -> impl IntoIterator<Item=&str>`
However, given `catalog_names` are only called when creating the information_schema views, I am not sure this optimization is warranted at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] seddonm1 commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602604745
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
Ok. Maybe we can find the standards document somewhere as I assume there is a fixed list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602914122
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
Review comment:
I guess I felt that `10` was a siimple as `0` -- lol though I don't really care
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r603345813
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -310,16 +324,25 @@ impl ExecutionContext {
name: impl Into<String>,
catalog: Arc<dyn CatalogProvider>,
) -> Option<Arc<dyn CatalogProvider>> {
- self.state
- .lock()
- .unwrap()
- .catalogs
- .insert(name.into(), catalog)
+ let name = name.into();
+
+ let state = self.state.lock().unwrap();
+ println!("Registering catalog for {}", name);
Review comment:
whoops -- good call -- will remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-809391146
Hey @jorgecarleitao, so I introduced the general concept in #9762 (design doc [here](https://docs.google.com/document/d/1_bCP_tjVRLJyOrMBOezSFNpF0hwPa1ZS_qMWv1uvtS4/edit#)), with a mix of rationales
Firstly, I saw Andrew's proposal for this info schema work and wanted to support namespacing this correctly, as it works in other DBs (we don't want to disallow people from registering a table called `tables`, for the sake of a silly example, and having that shadow the system one).
Secondly, in my use case (DataFusion as the execution engine for an actual DB), I have catalogs and schemas as a mechanism for logical organisation. From the proposal doc:
> - Multi-tenant software seeking more stringent data isolation might use catalogs to represent individual tenants
> - Schemas could be used to split tables by producer in a centralised data pipeline
> - In a data warehouse comprising both raw event logs and pre-aggregated summary tables, you could use “raw” and “summary” schemas to organise the tables by type
Your point about roles/permissions is totally correct - in my case currently, I'm dynamically building execution contexts based on tables that users are allowed to access according to my external metadata. Users can also create tables, which is again a metadata operation that informs future execution context creation (and these are created within a specific catalog and schema).
Lemme know if anything there is unclear, doing lunchbreak replies atm so may have missed something 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602590429
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -2048,6 +2076,130 @@ mod tests {
Ok(Arc::new(MemTable::try_new(schema, partitions)?))
}
+ #[tokio::test]
+ async fn information_schema_tables_not_exist_by_default() {
Review comment:
These tests illustrate how this feature works concisely I believe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-808556642
This looks awesome (and so hot on the heels of the basic catalog impl!) - will take a look tomorrow :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-809292029
Ok, I have addressed all comments and I think this PR is ready to merge (once it gets an approval).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r603229120
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -310,16 +324,25 @@ impl ExecutionContext {
name: impl Into<String>,
catalog: Arc<dyn CatalogProvider>,
) -> Option<Arc<dyn CatalogProvider>> {
- self.state
- .lock()
- .unwrap()
- .catalogs
- .insert(name.into(), catalog)
+ let name = name.into();
+
+ let state = self.state.lock().unwrap();
+ println!("Registering catalog for {}", name);
Review comment:
Leftover debugging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602704095
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -23,6 +23,67 @@ use std::any::Any;
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
+/// Represent a list of named catalogs
+pub trait CatalogList: Sync + Send {
+ /// Returns the catalog list as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Adds a new catalog to this catalog list
+ /// If a catalog of the same name existed before, it is replaced in the list and returned.
+ fn register_catalog(
+ &self,
+ name: String,
+ catalog: Arc<dyn CatalogProvider>,
+ ) -> Option<Arc<dyn CatalogProvider>>;
+
+ /// Retrieves the list of available catalog names
+ fn catalog_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific catalog by name, provided it exists.
+ fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>>;
+}
+
+/// Simple in-memory list of catalogs
+pub struct MemoryCatalogList {
+ /// Collection of catalogs containing schemas and ultimately TableProviders
+ pub catalogs: RwLock<HashMap<String, Arc<dyn CatalogProvider>>>,
+}
+
+impl MemoryCatalogList {
+ /// Instantiates a new MemoryDatabase with an empty collection of catalogs
Review comment:
Minor: MemoryCatalogList instead of MemoryDatabase here?
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -678,20 +695,31 @@ impl ExecutionContextState {
.resolve(&self.config.default_catalog, &self.config.default_schema)
}
+ fn catalog_for_ref<'a>(
+ &'a self,
+ resolved_ref: &ResolvedTableReference<'a>,
+ ) -> Result<Arc<dyn CatalogProvider>> {
+ if self.config.information_schema {
+ CatalogListWithInformationSchema::new(self.catalog_list.clone())
Review comment:
Not a massive issue as this isn't the critical path, but would it be worth doing this wrapping on catalog registration so we avoid query-side allocs here? I wonder if it's possible to remove the `CatalogListWithInformationSchema` entirely and just do this inside `register_catalog` instead. So if the info schema is enabled, wrap the catalog up in a `CatalogWithInformationSchema` before registering it with the list, or otherwise, register the plain catalog.
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -651,13 +662,19 @@ impl ExecutionConfig {
self.create_default_catalog_and_schema = create;
self
}
+
+ /// Enables the `information_schema` virtual tables
+ pub fn with_information_schema(mut self) -> Self {
+ self.information_schema = true;
Review comment:
Would it be worth enabling by default, even, and adding an option to _disable_ it here? I can't think of any downsides to enabling it, and for the subset of the info schema that we support, it's in line with the standard (as far as I'm aware without spending stupid money of a copy of the standard 😅)
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
+ schema_names: StringBuilder::new(10),
+ table_names: StringBuilder::new(10),
+ table_types: StringBuilder::new(10),
+ }
+ }
+
+ fn add_base_table(
+ &mut self,
+ catalog_name: impl AsRef<str>,
+ schema_name: impl AsRef<str>,
+ table_name: impl AsRef<str>,
+ ) {
+ // Note: append_value is actually infallable.
+ self.catalog_names
+ .append_value(catalog_name.as_ref())
+ .unwrap();
+ self.schema_names
+ .append_value(schema_name.as_ref())
+ .unwrap();
+ self.table_names.append_value(table_name.as_ref()).unwrap();
+ self.table_types.append_value("BASE TABLE").unwrap();
Review comment:
I could definitely see an argument that the csv/parquet TableProviders are temporary, but arbitrary TableProviders might use totally different semantics (e.g. in my case, fetching data from a persistent store that could be classed as a base table). If we want to support different options here, perhaps we need a new TableProvider method to indicate that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602613523
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -651,13 +662,19 @@ impl ExecutionConfig {
self.create_default_catalog_and_schema = create;
self
}
+
+ /// Enables the `information_schema` virtual tables
+ pub fn with_information_schema(mut self) -> Self {
+ self.information_schema = true;
Review comment:
Can it be disabled as well - or don't we expect that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] seddonm1 commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
seddonm1 commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602597882
##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+ array::StringBuilder,
+ datatypes::{DataType, Field, Schema},
+ record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+ catalog::{CatalogList, CatalogProvider},
+ schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+ inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+ pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+ Self { inner }
+ }
+
+ /// Returns a catalog provider that also contains the information schema table provider.
+ pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+ self.inner.catalog(name).map(|catalog| {
+ let catalog_list = self.inner.clone();
+
+ Arc::new(CatalogWithInformationSchema {
+ catalog_list,
+ inner: catalog,
+ }) as Arc<dyn CatalogProvider>
+ })
+ }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+ catalog_list: Arc<dyn CatalogList>,
+ /// wrapped provider
+ inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+ fn as_any(&self) -> &dyn any::Any {
+ self
+ }
+
+ fn schema_names(&self) -> Vec<String> {
+ self.inner
+ .schema_names()
+ .into_iter()
+ .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+ .collect::<Vec<String>>()
+ }
+
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+ if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+ Some(Arc::new(InformationSchemaProvider {
+ catalog_list: self.catalog_list.clone(),
+ }))
+ } else {
+ self.inner.schema(name)
+ }
+ }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+ catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+ fn as_any(&self) -> &(dyn any::Any + 'static) {
+ self
+ }
+
+ fn table_names(&self) -> Vec<String> {
+ vec![TABLES.to_string()]
+ }
+
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+ if name.eq_ignore_ascii_case("tables") {
+ // create a mem table with the names of tables
+ let mut builder = InformationSchemaTablesBuilder::new();
+
+ for catalog_name in self.catalog_list.catalog_names() {
+ let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+ for schema_name in catalog.schema_names() {
+ let schema = catalog.schema(&schema_name).unwrap();
+ for table_name in schema.table_names() {
+ builder.add_base_table(&catalog_name, &schema_name, table_name)
+ }
+ }
+
+ // Add a final list for the information schema tables themselves
+ builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+ }
+
+ let mem_table = builder.build();
+
+ Some(Arc::new(mem_table))
+ } else {
+ None
+ }
+ }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+ catalog_names: StringBuilder,
+ schema_names: StringBuilder,
+ table_names: StringBuilder,
+ table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+ fn new() -> Self {
+ Self {
+ catalog_names: StringBuilder::new(10),
Review comment:
why 10?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString edited a comment on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString edited a comment on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-809391146
Hey @jorgecarleitao, so I introduced the general concept in #9762 (design doc [here](https://docs.google.com/document/d/1_bCP_tjVRLJyOrMBOezSFNpF0hwPa1ZS_qMWv1uvtS4/edit#)), with a mix of rationales
Firstly, I saw Andrew's proposal for this info schema work and wanted to support namespacing this correctly, as it works in other DBs (we don't want to disallow people from registering a table called `tables`, for the sake of a silly example, and having that shadow the system one).
Secondly, in my use case (DataFusion as the execution engine for an actual DB), I have catalogs and schemas as a mechanism for logical organisation. From the proposal doc:
> - Multi-tenant software seeking more stringent data isolation might use catalogs to represent individual tenants
> - Schemas could be used to split tables by producer in a centralised data pipeline
> - In a data warehouse comprising both raw event logs and pre-aggregated summary tables, you could use “raw” and “summary” schemas to organise the tables by type
Your point about roles/permissions is totally correct - in my case currently, I'm dynamically building execution contexts based on tables that users are allowed to access according to my external metadata. Users can also create tables, which is again a metadata operation that informs future execution context creation (and these are created within a specific catalog and schema).
Lemme know if anything there is unclear, doing lunchbreak replies atm so may have missed something 😄
Edit: oh, and one of the design goals was to ensure that as little as possible changed for use cases that don't care about namespacing - if a singular flat namespace works for a given use case, nothing user-facing should need to change, and you'll just see a default catalog/schema of "datafusion.public" when querying the info schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602904515
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -651,13 +662,19 @@ impl ExecutionConfig {
self.create_default_catalog_and_schema = create;
self
}
+
+ /// Enables the `information_schema` virtual tables
+ pub fn with_information_schema(mut self) -> Self {
+ self.information_schema = true;
Review comment:
> Can it be disabled as well - or don't we expect that?
I didn't expect people to want to be able to change the setting at runtime on the same `Context` -- either they would want the information schema or not. However, for consistency with the other settings I think it would be good to allow the flag to be passed in and I will do so
> Would it be worth enabling by default, even, and adding an option to disable it here?
There might be some small runtime overhead for always enabling it. I don't have any strong preference on one way or the other. I am happy to change the default. Does anyone else have any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r603386970
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -310,16 +324,25 @@ impl ExecutionContext {
name: impl Into<String>,
catalog: Arc<dyn CatalogProvider>,
) -> Option<Arc<dyn CatalogProvider>> {
- self.state
- .lock()
- .unwrap()
- .catalogs
- .insert(name.into(), catalog)
+ let name = name.into();
+
+ let state = self.state.lock().unwrap();
+ println!("Registering catalog for {}", name);
Review comment:
might be a good one for a `debug!` line actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#issuecomment-809319874
I've also logged https://issues.apache.org/jira/browse/ARROW-12132 for the table provider "type" followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org