[GitHub] [arrow] returnString commented on a change in pull request #9818: ARROW-12106: [Rust][DataFusion] Support `SELECT * from information_schema.tables`

[GitBox <gi...@apache.org>]

returnString commented on a change in pull request #9818:
URL: https://github.com/apache/arrow/pull/9818#discussion_r602704095



##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -23,6 +23,67 @@ use std::any::Any;
 use std::collections::HashMap;
 use std::sync::{Arc, RwLock};
 
+/// Represent a list of named catalogs
+pub trait CatalogList: Sync + Send {
+    /// Returns the catalog list as [`Any`](std::any::Any)
+    /// so that it can be downcast to a specific implementation.
+    fn as_any(&self) -> &dyn Any;
+
+    /// Adds a new catalog to this catalog list
+    /// If a catalog of the same name existed before, it is replaced in the list and returned.
+    fn register_catalog(
+        &self,
+        name: String,
+        catalog: Arc<dyn CatalogProvider>,
+    ) -> Option<Arc<dyn CatalogProvider>>;
+
+    /// Retrieves the list of available catalog names
+    fn catalog_names(&self) -> Vec<String>;
+
+    /// Retrieves a specific catalog by name, provided it exists.
+    fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>>;
+}
+
+/// Simple in-memory list of catalogs
+pub struct MemoryCatalogList {
+    /// Collection of catalogs containing schemas and ultimately TableProviders
+    pub catalogs: RwLock<HashMap<String, Arc<dyn CatalogProvider>>>,
+}
+
+impl MemoryCatalogList {
+    /// Instantiates a new MemoryDatabase with an empty collection of catalogs

Review comment:
       Minor: MemoryCatalogList instead of MemoryDatabase here?

##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -678,20 +695,31 @@ impl ExecutionContextState {
             .resolve(&self.config.default_catalog, &self.config.default_schema)
     }
 
+    fn catalog_for_ref<'a>(
+        &'a self,
+        resolved_ref: &ResolvedTableReference<'a>,
+    ) -> Result<Arc<dyn CatalogProvider>> {
+        if self.config.information_schema {
+            CatalogListWithInformationSchema::new(self.catalog_list.clone())

Review comment:
       Not a massive issue as this isn't the critical path, but would it be worth doing this wrapping on catalog registration so we avoid query-side allocs here? I wonder if it's possible to remove the `CatalogListWithInformationSchema` entirely and just do this inside `register_catalog` instead. So if the info schema is enabled, wrap the catalog up in a `CatalogWithInformationSchema` before registering it with the list, or otherwise, register the plain catalog.

##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -651,13 +662,19 @@ impl ExecutionConfig {
         self.create_default_catalog_and_schema = create;
         self
     }
+
+    /// Enables the `information_schema` virtual tables
+    pub fn with_information_schema(mut self) -> Self {
+        self.information_schema = true;

Review comment:
       Would it be worth enabling by default, even, and adding an option to _disable_ it here? I can't think of any downsides to enabling it, and for the subset of the info schema that we support, it's in line with the standard (as far as I'm aware without spending stupid money of a copy of the standard 😅)

##########
File path: rust/datafusion/src/catalog/information_schema.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implements the SQL [Information Schema] for DataFusion.
+//!
+//! Information Schema](https://en.wikipedia.org/wiki/Information_schema)
+
+use std::{any, sync::Arc};
+
+use arrow::{
+    array::StringBuilder,
+    datatypes::{DataType, Field, Schema},
+    record_batch::RecordBatch,
+};
+
+use crate::datasource::{MemTable, TableProvider};
+
+use super::{
+    catalog::{CatalogList, CatalogProvider},
+    schema::SchemaProvider,
+};
+
+const INFORMATION_SCHEMA: &str = "information_schema";
+const TABLES: &str = "tables";
+
+/// Wraps a [`CatalogList`] so that it also provides an
+/// `information_schema` schema view as well in each catalog.
+pub(crate) struct CatalogListWithInformationSchema {
+    inner: Arc<dyn CatalogList>,
+}
+
+impl CatalogListWithInformationSchema {
+    pub(crate) fn new(inner: Arc<dyn CatalogList>) -> Self {
+        Self { inner }
+    }
+
+    /// Returns a catalog provider that also contains the information schema table provider.
+    pub(crate) fn catalog(&self, name: &str) -> Option<Arc<dyn CatalogProvider>> {
+        self.inner.catalog(name).map(|catalog| {
+            let catalog_list = self.inner.clone();
+
+            Arc::new(CatalogWithInformationSchema {
+                catalog_list,
+                inner: catalog,
+            }) as Arc<dyn CatalogProvider>
+        })
+    }
+}
+
+/// Wraps another [`CatalogProvider`] and adds a "information_schema"
+/// schema that can introspect on tables in the catalog_list
+struct CatalogWithInformationSchema {
+    catalog_list: Arc<dyn CatalogList>,
+    /// wrapped provider
+    inner: Arc<dyn CatalogProvider>,
+}
+
+impl CatalogProvider for CatalogWithInformationSchema {
+    fn as_any(&self) -> &dyn any::Any {
+        self
+    }
+
+    fn schema_names(&self) -> Vec<String> {
+        self.inner
+            .schema_names()
+            .into_iter()
+            .chain(std::iter::once(INFORMATION_SCHEMA.to_string()))
+            .collect::<Vec<String>>()
+    }
+
+    fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
+        if name.eq_ignore_ascii_case(INFORMATION_SCHEMA) {
+            Some(Arc::new(InformationSchemaProvider {
+                catalog_list: self.catalog_list.clone(),
+            }))
+        } else {
+            self.inner.schema(name)
+        }
+    }
+}
+
+/// Implements the `information_schema` virtual schema and tables
+///
+/// The underlying tables in the `information_schema` are created on
+/// demand. This means that if more tables are added to the underlying
+/// providers, they will appear the next time the `information_schema`
+/// table is queried.
+struct InformationSchemaProvider {
+    catalog_list: Arc<dyn CatalogList>,
+}
+
+impl SchemaProvider for InformationSchemaProvider {
+    fn as_any(&self) -> &(dyn any::Any + 'static) {
+        self
+    }
+
+    fn table_names(&self) -> Vec<String> {
+        vec![TABLES.to_string()]
+    }
+
+    fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>> {
+        if name.eq_ignore_ascii_case("tables") {
+            // create a mem table with the names of tables
+            let mut builder = InformationSchemaTablesBuilder::new();
+
+            for catalog_name in self.catalog_list.catalog_names() {
+                let catalog = self.catalog_list.catalog(&catalog_name).unwrap();
+
+                for schema_name in catalog.schema_names() {
+                    let schema = catalog.schema(&schema_name).unwrap();
+                    for table_name in schema.table_names() {
+                        builder.add_base_table(&catalog_name, &schema_name, table_name)
+                    }
+                }
+
+                // Add a final list for the information schema tables themselves
+                builder.add_system_table(&catalog_name, INFORMATION_SCHEMA, TABLES);
+            }
+
+            let mem_table = builder.build();
+
+            Some(Arc::new(mem_table))
+        } else {
+            None
+        }
+    }
+}
+
+/// Builds the `information_schema.TABLE` table row by row
+
+struct InformationSchemaTablesBuilder {
+    catalog_names: StringBuilder,
+    schema_names: StringBuilder,
+    table_names: StringBuilder,
+    table_types: StringBuilder,
+}
+
+impl InformationSchemaTablesBuilder {
+    fn new() -> Self {
+        Self {
+            catalog_names: StringBuilder::new(10),
+            schema_names: StringBuilder::new(10),
+            table_names: StringBuilder::new(10),
+            table_types: StringBuilder::new(10),
+        }
+    }
+
+    fn add_base_table(
+        &mut self,
+        catalog_name: impl AsRef<str>,
+        schema_name: impl AsRef<str>,
+        table_name: impl AsRef<str>,
+    ) {
+        // Note: append_value is actually infallable.
+        self.catalog_names
+            .append_value(catalog_name.as_ref())
+            .unwrap();
+        self.schema_names
+            .append_value(schema_name.as_ref())
+            .unwrap();
+        self.table_names.append_value(table_name.as_ref()).unwrap();
+        self.table_types.append_value("BASE TABLE").unwrap();

Review comment:
       I could definitely see an argument that the csv/parquet TableProviders are temporary, but arbitrary TableProviders might use totally different semantics (e.g. in my case, fetching data from a persistent store that could be classed as a base table). If we want to support different options here, perhaps we need a new TableProvider method to indicate that?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org