You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/21 12:30:15 UTC
[GitHub] [arrow] returnString opened a new pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
returnString opened a new pull request #9762:
URL: https://github.com/apache/arrow/pull/9762
This is an implementation of catalog and schema providers to support table namespacing (see the [design doc](https://docs.google.com/document/d/1_bCP_tjVRLJyOrMBOezSFNpF0hwPa1ZS_qMWv1uvtS4/edit?usp=sharing)).
I'm creating this draft PR as a supporting implementation for the proposal, to prove out that the work can be done whilst minimising API churn and still allowing for use cases that don't care at all about the notion of catalogs or schemas; in this new setup, the default namespace is `dafafusion.public`, which will be created automatically with the default execution context config and allow for table registration.
## Highlights
- Datasource map removed in execution context state, replaced with catalog map
- Execution context allows for registering new catalog providers
- Catalog providers can be queried for their constituent schema providers
- Schema providers can be queried for table providers, similarly to the old datasource map
- Includes basic implementations of `CatalogProvider` and `SchemaProvider` backed by hashmaps
- New `TableReference` enum maps to various ways of referring to a table in sql
- Bare: `my_table`
- Partial: `schema.my_table`
- Full: `catalog.schema.my_table`
- Given a default catalog and schema, `TableReference` instances of any variant can be converted to a `ResolvedTableReference`, which always include all three components
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9762:
URL: https://github.com/apache/arrow/pull/9762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on a change in pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on a change in pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#discussion_r599044375
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of catalogs,
+//! representing collections of named schemas.
+
+use crate::catalog::schema::SchemaProvider;
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a catalog, comprising a number of named schemas.
+pub trait CatalogProvider: Sync + Send {
+ /// Returns the catalog provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available schema names in this catalog.
+ fn schema_names(&self) -> Vec<String>;
Review comment:
I iniitally tried to implement these returning `&[&str]` but with the threading requirements I struggled to get anything working that way. I _think_ this would require a larger refactoring to enable, but I'm not 100% sure on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803661931
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=h1) Report
> Merging [#9762](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=desc) (66de8d3) into [master](https://codecov.io/gh/apache/arrow/commit/29feea07fea118ad6ffc4d2615a5798412fe2f19?el=desc) (29feea0) will **decrease** coverage by `0.00%`.
> The diff coverage is `80.10%`.
[](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9762 +/- ##
==========================================
- Coverage 82.59% 82.59% -0.01%
==========================================
Files 248 252 +4
Lines 58294 58983 +689
==========================================
+ Hits 48149 48716 +567
- Misses 10145 10267 +122
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/examples/flight\_server.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL2V4YW1wbGVzL2ZsaWdodF9zZXJ2ZXIucnM=) | `0.00% <0.00%> (ø)` | |
| [rust/datafusion/examples/simple\_udaf.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL2V4YW1wbGVzL3NpbXBsZV91ZGFmLnJz) | `0.00% <ø> (ø)` | |
| [rust/datafusion/src/datasource/datasource.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9kYXRhc291cmNlL2RhdGFzb3VyY2UucnM=) | `100.00% <ø> (ø)` | |
| [rust/datafusion/src/datasource/memory.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9kYXRhc291cmNlL21lbW9yeS5ycw==) | `85.15% <ø> (ø)` | |
| [rust/datafusion/src/execution/dataframe\_impl.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9leGVjdXRpb24vZGF0YWZyYW1lX2ltcGwucnM=) | `89.10% <0.00%> (-1.03%)` | :arrow_down: |
| [rust/datafusion/src/optimizer/constant\_folding.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvY29uc3RhbnRfZm9sZGluZy5ycw==) | `92.30% <0.00%> (-0.36%)` | :arrow_down: |
| [...datafusion/src/optimizer/hash\_build\_probe\_order.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvaGFzaF9idWlsZF9wcm9iZV9vcmRlci5ycw==) | `53.60% <0.00%> (-1.72%)` | :arrow_down: |
| [...t/datafusion/src/optimizer/projection\_push\_down.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvcHJvamVjdGlvbl9wdXNoX2Rvd24ucnM=) | `98.66% <ø> (ø)` | |
| [rust/datafusion/src/physical\_plan/mod.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL21vZC5ycw==) | `88.00% <ø> (ø)` | |
| [rust/parquet/src/basic.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9wYXJxdWV0L3NyYy9iYXNpYy5ycw==) | `88.59% <ø> (+1.36%)` | :arrow_up: |
| ... and [35 more](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=footer). Last update [81d6724...66de8d3](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io edited a comment on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803661931
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=h1) Report
> Merging [#9762](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=desc) (03048c6) into [master](https://codecov.io/gh/apache/arrow/commit/29feea07fea118ad6ffc4d2615a5798412fe2f19?el=desc) (29feea0) will **increase** coverage by `0.00%`.
> The diff coverage is `80.36%`.
[](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9762 +/- ##
========================================
Coverage 82.59% 82.59%
========================================
Files 248 252 +4
Lines 58294 59004 +710
========================================
+ Hits 48149 48737 +588
- Misses 10145 10267 +122
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/examples/flight\_server.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL2V4YW1wbGVzL2ZsaWdodF9zZXJ2ZXIucnM=) | `0.00% <0.00%> (ø)` | |
| [rust/datafusion/examples/simple\_udaf.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL2V4YW1wbGVzL3NpbXBsZV91ZGFmLnJz) | `0.00% <ø> (ø)` | |
| [rust/datafusion/src/datasource/datasource.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9kYXRhc291cmNlL2RhdGFzb3VyY2UucnM=) | `100.00% <ø> (ø)` | |
| [rust/datafusion/src/datasource/memory.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9kYXRhc291cmNlL21lbW9yeS5ycw==) | `85.15% <ø> (ø)` | |
| [rust/datafusion/src/execution/dataframe\_impl.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9leGVjdXRpb24vZGF0YWZyYW1lX2ltcGwucnM=) | `89.10% <0.00%> (-1.03%)` | :arrow_down: |
| [rust/datafusion/src/optimizer/constant\_folding.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvY29uc3RhbnRfZm9sZGluZy5ycw==) | `92.30% <0.00%> (-0.36%)` | :arrow_down: |
| [...datafusion/src/optimizer/hash\_build\_probe\_order.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvaGFzaF9idWlsZF9wcm9iZV9vcmRlci5ycw==) | `53.60% <0.00%> (-1.72%)` | :arrow_down: |
| [...t/datafusion/src/optimizer/projection\_push\_down.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvcHJvamVjdGlvbl9wdXNoX2Rvd24ucnM=) | `98.66% <ø> (ø)` | |
| [rust/datafusion/src/physical\_plan/mod.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL21vZC5ycw==) | `88.00% <ø> (ø)` | |
| [rust/parquet/src/basic.rs](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree#diff-cnVzdC9wYXJxdWV0L3NyYy9iYXNpYy5ycw==) | `88.59% <ø> (+1.36%)` | :arrow_up: |
| ... and [35 more](https://codecov.io/gh/apache/arrow/pull/9762/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=footer). Last update [81d6724...03048c6](https://codecov.io/gh/apache/arrow/pull/9762?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString edited a comment on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString edited a comment on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803626826
Thanks both :)
> I noted a couple of unwraps in the code. It would be good to document why they are safe or consider having those methods return Result if they are not safe.
I've just audited all the unwraps that are introduced in these commits, and they fall into two categories:
- test cases that aren't already set up to return `Result<()>`
- unwrapping guard objects for `Mutex`/`RwLock` - afaik these only error if the lock is poisoned and this usage is consistent with existing lock usage e.g. state access in the `ExecutionContext`
> Apart from some tests & remaining cleanup (e.g. of the "quick hack") I really like the direction this is going!
I think the quick hack (context: retrieving a list of table names from the execution ctx, assuming a flat namespace) can probably be removed outright, as `ctx.catalog("my_db")?.schema("my_schema")?.table_names()` would be a much more explicit way of doing this going forward. As far as I can tell, `ExecutionContext::tables` isn't referenced at all in the codebase internally, but this would introduce a bit more API breakage.
As for tests, yes, the coverage so far is basically "didn't break anything" and "all three types of reference to a given table in the default catalog/schema resolve correctly", so I'll try and plan out some more in-depth testing covering e.g. tables of the same name in multiple schemas/catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-804925070
@returnString do you plan anything else for this PR? I think it is ready to merge but I wanted to check to see if you think there is anything else left to do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803569847
https://issues.apache.org/jira/browse/ARROW-12037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-805304397
Thanks again @returnString ! This is a great feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803620661
This is truly awesome @returnString !
Apart from some tests & remaining cleanup (e.g. of the "quick hack") I really like the direction this is going!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803620665
Thanks @returnString and I appreciate the well-written design doc to explain the PR. I didn't go through the code in great detail but I like the design. I noted a couple of unwraps in the code. It would be good to document why they are safe or consider having those methods return `Result` if they are not safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on a change in pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on a change in pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#discussion_r599063974
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -512,6 +573,12 @@ pub struct ExecutionConfig {
optimizers: Vec<Arc<dyn OptimizerRule + Send + Sync>>,
/// Responsible for planning `LogicalPlan`s, and `ExecutionPlan`
query_planner: Arc<dyn QueryPlanner + Send + Sync>,
+ /// Default catalog name for table resolution
+ default_catalog: String,
Review comment:
This is a very good question!
I think we could allow for that distinction, but it gets a little bit messy as we'd probably want to enforce that you can't have a default schema without a default catalog, so it'd look something like `Option<(String, Option<String>)>` (in fact, I had something like that for a bit before I settled on just using `datafusion.public` as the default setting for back-compat).
I'm struggling to think of a use case though 🤔 Maybe if someone were developing a system where they wanted to _force_ users to fully qualify their table names?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb removed a comment on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb removed a comment on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-804027173
I plan to review this PR later today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-804935745
> @returnString do you plan anything else for this PR? I think it is ready to merge but I wanted to check to see if you think there is anything else left to do?
I was looking to see if I could improve the test setup any further, but on reflection I'm actually pretty happy with the state of things; I don't think we're missing any use cases in the coverage (at least, ones I can conceive of right now). So from my side, happy with things as they are :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803626826
Thanks both :)
> I noted a couple of unwraps in the code. It would be good to document why they are safe or consider having those methods return Result if they are not safe.
I've just audited all the unwraps that are introduced in these commits, and they fall into two categories:
- test cases that aren't already set up to return `Result<()>`
- unwrapping guard objects for `Mutex`/`RwLock` - afaik these only error if the lock is poisoned and this usage is consistent with existing lock usage e.g. state access in the `ExecutionContext`
> Apart from some tests & remaining cleanup (e.g. of the "quick hack") I really like the direction this is going!
I think the quick hack (context: retrieving a list of table names from the execution ctx, assuming a flat namespace) can probably be removed outright, as `ctx.catalog("my_db")?.schema("my_schema")?.table_names()` would be a much more explicit way of doing this going forward. As far as I can tell, it's not referenced at all in the codebase internally, but this would introduce a bit more API breakage.
As for tests, yes, the coverage so far is basically "didn't break anything" and "all three types of reference to a given table in the default catalog/schema resolve correctly", so I'll try and plan out some more in-depth testing covering e.g. tables of the same name in multiple schemas/catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-804027020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#issuecomment-803669166
Whilst I still want to try and conjure up a few more test cases, I think this PR is now in a state where I'm happy to receive actual code reviews, so I've removed the draft status :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] returnString commented on a change in pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
returnString commented on a change in pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#discussion_r599049329
##########
File path: rust/datafusion/src/catalog/schema.rs
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of schemas,
+//! representing collections of named tables.
+
+use crate::datasource::TableProvider;
+use crate::error::{DataFusionError, Result};
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a schema, comprising a number of named tables.
+pub trait SchemaProvider: Sync + Send {
+ /// Returns the schema provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available table names in this schema.
+ fn table_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific table from the schema by name, provided it exists.
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>>;
+
+ /// If supported by the implementation, adds a new table to this schema.
+ /// If a table of the same name existed before, it is replaced in the schema and returned.
+ #[allow(unused_variables)]
+ fn register_table(
+ &self,
+ name: String,
+ table: Arc<dyn TableProvider>,
Review comment:
The underscore method was my first approach here but I then discovered that it carried those symbols through to the docs as well (e.g. inline parameter names in vscode using rust analyser - presumably also the generated ones in docs.rs too? didn't test that bit)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#discussion_r599086894
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of catalogs,
+//! representing collections of named schemas.
+
+use crate::catalog::schema::SchemaProvider;
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a catalog, comprising a number of named schemas.
+pub trait CatalogProvider: Sync + Send {
+ /// Returns the catalog provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available schema names in this catalog.
+ fn schema_names(&self) -> Vec<String>;
Review comment:
Vec<&str> might be possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9762: ARROW-12037: [Rust] [DataFusion] Support catalogs and schemas for table namespacing
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9762:
URL: https://github.com/apache/arrow/pull/9762#discussion_r599027565
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of catalogs,
+//! representing collections of named schemas.
+
+use crate::catalog::schema::SchemaProvider;
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a catalog, comprising a number of named schemas.
+pub trait CatalogProvider: Sync + Send {
+ /// Returns the catalog provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available schema names in this catalog.
+ fn schema_names(&self) -> Vec<String>;
Review comment:
What do you think about returning something more like `Vec<&str>` to prevent requiring a copy?
Ideally it would be nice if we could do something like
```
fn schema_names(&self) -> impl Iterator<Item=&str>
```
As all uses of the results here will need to iterate over the names I suspect
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of catalogs,
+//! representing collections of named schemas.
+
+use crate::catalog::schema::SchemaProvider;
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a catalog, comprising a number of named schemas.
+pub trait CatalogProvider: Sync + Send {
+ /// Returns the catalog provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available schema names in this catalog.
+ fn schema_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific schema from the catalog by name, provided it exists.
+ fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>>;
+}
+
+/// Simple in-memory implementation of a catalog.
+pub struct MemoryCatalogProvider {
Review comment:
In some other PR perhaps we can put the concrete implementations into their own modules. I don't think this one is big enough to warrant that yet, however; I just wanted to point it out
##########
File path: rust/datafusion/src/catalog/schema.rs
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of schemas,
+//! representing collections of named tables.
+
+use crate::datasource::TableProvider;
+use crate::error::{DataFusionError, Result};
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a schema, comprising a number of named tables.
+pub trait SchemaProvider: Sync + Send {
+ /// Returns the schema provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available table names in this schema.
+ fn table_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific table from the schema by name, provided it exists.
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>>;
+
+ /// If supported by the implementation, adds a new table to this schema.
+ /// If a table of the same name existed before, it is replaced in the schema and returned.
+ #[allow(unused_variables)]
+ fn register_table(
+ &self,
+ name: String,
+ table: Arc<dyn TableProvider>,
+ ) -> Result<Option<Arc<dyn TableProvider>>> {
+ Err(DataFusionError::Execution(
+ "schema provider does not support registering tables".to_owned(),
+ ))
+ }
+
+ /// If supported by the implementation, removes an existing table from this schema and returns it.
+ /// If no table of that name exists, returns Ok(None).
+ #[allow(unused_variables)]
+ fn deregister_table(&self, name: &str) -> Result<Option<Arc<dyn TableProvider>>> {
Review comment:
```suggestion
fn deregister_table(&self, _name: &str) -> Result<Option<Arc<dyn TableProvider>>> {
```
##########
File path: rust/datafusion/src/catalog/schema.rs
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of schemas,
+//! representing collections of named tables.
+
+use crate::datasource::TableProvider;
+use crate::error::{DataFusionError, Result};
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a schema, comprising a number of named tables.
+pub trait SchemaProvider: Sync + Send {
+ /// Returns the schema provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available table names in this schema.
+ fn table_names(&self) -> Vec<String>;
Review comment:
same suggestion here to try and avoid copying strings if we can avoid it
##########
File path: rust/datafusion/tests/sql.rs
##########
@@ -2504,3 +2507,36 @@ async fn inner_join_qualified_names() -> Result<()> {
}
Ok(())
}
+
+#[tokio::test]
+async fn qualified_table_references() -> Result<()> {
+ let mut ctx = ExecutionContext::new();
+ register_aggregate_csv(&mut ctx)?;
+
+ for table_ref in &[
+ "aggregate_test_100",
+ "public.aggregate_test_100",
+ "datafusion.public.aggregate_test_100",
+ ] {
+ let sql = format!("SELECT COUNT(*) FROM {}", table_ref);
+ let results = execute(&mut ctx, &sql).await;
+ assert_eq!(results, vec![vec!["100"]]);
+ }
+ Ok(())
+}
+
+#[tokio::test]
+async fn invalid_qualified_table_references() -> Result<()> {
+ let mut ctx = ExecutionContext::new();
+ register_aggregate_csv(&mut ctx)?;
+
+ for table_ref in &[
+ "nonexistentschema.aggregate_test_100",
+ "nonexistentcatalog.public.aggregate_test_100",
+ "way.too.many.namespaces.as.ident.prefixes.aggregate_test_100",
Review comment:
😆
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -564,13 +634,30 @@ impl ExecutionConfig {
self.optimizers.push(optimizer_rule);
self
}
+
+ /// Selects a name for the default catalog and schema
+ pub fn with_default_catalog_and_schema(
+ mut self,
+ catalog: impl Into<String>,
+ schema: impl Into<String>,
+ ) -> Self {
+ self.default_catalog = catalog.into();
+ self.default_schema = schema.into();
+ self
+ }
+
+ /// Controls whether the default catalog and schema will be automatically created
+ pub fn create_default_catalog_and_schema(mut self, create: bool) -> Self {
+ self.create_default_catalog_and_schema = create;
+ self
+ }
}
/// Execution context for registering data sources and executing queries
#[derive(Clone)]
pub struct ExecutionContextState {
- /// Data sources that are registered with the context
- pub datasources: HashMap<String, Arc<dyn TableProvider + Send + Sync>>,
+ /// Collection of catalogs containing schemas and ultimately tables
Review comment:
```suggestion
/// Collection of catalogs containing schemas and ultimately TableProviders
```
##########
File path: rust/datafusion/src/catalog/mod.rs
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This module contains interfaces and default implementations
+//! of table namespacing concepts, including catalogs and schemas.
+
+pub mod catalog;
+pub mod schema;
+
+use crate::error::DataFusionError;
+use std::convert::TryFrom;
+
+/// Represents a resolved path to a table of the form "catalog.schema.table"
+#[derive(Clone, Copy)]
+pub struct ResolvedTableReference<'a> {
+ /// The catalog (aka database) containing the table
+ pub catalog: &'a str,
+ /// The schema containing the table
+ pub schema: &'a str,
+ /// The table name
+ pub table: &'a str,
+}
+
+/// Represents a path to a table that may require further resolution
+#[derive(Clone, Copy)]
+pub enum TableReference<'a> {
+ /// An unqualified table reference, e.g. "table"
+ Bare {
+ /// The table name
+ table: &'a str,
+ },
+ /// A partially resolved table reference, e.g. "schema.table"
+ Partial {
+ /// The schema containing the table
+ schema: &'a str,
+ /// The table name
+ table: &'a str,
+ },
+ /// A fully resolved table reference, e.g. "catalog.schema.table"
+ Full {
+ /// The catalog (aka database) containing the table
+ catalog: &'a str,
+ /// The schema containing the table
+ schema: &'a str,
+ /// The table name
+ table: &'a str,
+ },
+}
+
+impl<'a> TableReference<'a> {
+ /// Retrieve the actual table name, regardless of qualification
+ pub fn table(&self) -> &str {
+ match self {
+ Self::Full { table, .. }
+ | Self::Partial { table, .. }
+ | Self::Bare { table } => table,
+ }
+ }
+
+ /// Given a default catalog and schema, ensure this table reference is fully resolved
+ pub fn resolve(
+ self,
+ default_catalog: &'a str,
+ default_schema: &'a str,
+ ) -> ResolvedTableReference<'a> {
+ match self {
+ Self::Full {
+ catalog,
+ schema,
+ table,
+ } => ResolvedTableReference {
+ catalog,
+ schema,
+ table,
+ },
+ Self::Partial { schema, table } => ResolvedTableReference {
+ catalog: default_catalog,
+ schema,
+ table,
+ },
+ Self::Bare { table } => ResolvedTableReference {
+ catalog: default_catalog,
+ schema: default_schema,
+ table,
+ },
+ }
+ }
+}
+
+impl<'a> From<&'a str> for TableReference<'a> {
+ fn from(s: &'a str) -> Self {
+ Self::Bare { table: s }
+ }
+}
+
+impl<'a> From<ResolvedTableReference<'a>> for TableReference<'a> {
+ fn from(resolved: ResolvedTableReference<'a>) -> Self {
+ Self::Full {
+ catalog: resolved.catalog,
+ schema: resolved.schema,
+ table: resolved.table,
+ }
+ }
+}
+
+impl<'a> TryFrom<&'a sqlparser::ast::ObjectName> for TableReference<'a> {
+ type Error = DataFusionError;
+
+ fn try_from(value: &'a sqlparser::ast::ObjectName) -> Result<Self, Self::Error> {
Review comment:
👍
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -512,6 +573,12 @@ pub struct ExecutionConfig {
optimizers: Vec<Arc<dyn OptimizerRule + Send + Sync>>,
/// Responsible for planning `LogicalPlan`s, and `ExecutionPlan`
query_planner: Arc<dyn QueryPlanner + Send + Sync>,
+ /// Default catalog name for table resolution
+ default_catalog: String,
Review comment:
Would we ever need to distinguish between `""` and `None`? If so, perhaps we can make this field `default_catalog: Option<String>` instead
##########
File path: rust/datafusion/src/catalog/catalog.rs
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of catalogs,
Review comment:
👍
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1922,6 +2035,114 @@ mod tests {
Ok(())
}
Review comment:
these are excellent tests . 👍
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1922,6 +2035,114 @@ mod tests {
Ok(())
}
+ fn table_with_sequence(
+ seq_start: i32,
+ seq_end: i32,
+ ) -> Result<Arc<dyn TableProvider>> {
+ let schema = Arc::new(Schema::new(vec![Field::new("i", DataType::Int32, true)]));
+ let arr = Arc::new(Int32Array::from((seq_start..=seq_end).collect::<Vec<_>>()));
+ let partitions = vec![vec![RecordBatch::try_new(
+ schema.clone(),
+ vec![arr as ArrayRef],
+ )?]];
+ Ok(Arc::new(MemTable::try_new(schema, partitions)?))
+ }
+
+ #[tokio::test]
+ async fn disabled_default_catalog_and_schema() -> Result<()> {
+ let mut ctx = ExecutionContext::with_config(
+ ExecutionConfig::new().create_default_catalog_and_schema(false),
+ );
+
+ assert!(matches!(
+ ctx.register_table("test", table_with_sequence(1, 1)?),
+ Err(DataFusionError::Plan(_))
+ ));
+
+ assert!(matches!(
+ ctx.sql("select * from datafusion.public.test"),
+ Err(DataFusionError::Plan(_))
+ ));
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn custom_catalog_and_schema() -> Result<()> {
+ let mut ctx = ExecutionContext::with_config(
+ ExecutionConfig::new()
+ .create_default_catalog_and_schema(false)
+ .with_default_catalog_and_schema("my_catalog", "my_schema"),
+ );
+
+ let catalog = MemoryCatalogProvider::new();
+ let schema = MemorySchemaProvider::new();
+ schema.register_table("test".to_owned(), table_with_sequence(1, 1)?)?;
+ catalog.register_schema("my_schema", Arc::new(schema));
+ ctx.register_catalog("my_catalog", Arc::new(catalog));
+
+ for table_ref in &["my_catalog.my_schema.test", "my_schema.test", "test"] {
+ let result = plan_and_collect(
+ &mut ctx,
+ &format!("SELECT COUNT(*) AS count FROM {}", table_ref),
+ )
+ .await?;
+
+ let expected = vec![
+ "+-------+",
+ "| count |",
+ "+-------+",
+ "| 1 |",
+ "+-------+",
+ ];
+ assert_batches_eq!(expected, &result);
+ }
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn cross_catalog_access() -> Result<()> {
+ let mut ctx = ExecutionContext::new();
+
+ let catalog_a = MemoryCatalogProvider::new();
+ let schema_a = MemorySchemaProvider::new();
+ schema_a.register_table("table_a".to_owned(), table_with_sequence(1, 1)?)?;
+ catalog_a.register_schema("schema_a", Arc::new(schema_a));
+ ctx.register_catalog("catalog_a", Arc::new(catalog_a));
+
+ let catalog_b = MemoryCatalogProvider::new();
+ let schema_b = MemorySchemaProvider::new();
+ schema_b.register_table("table_b".to_owned(), table_with_sequence(1, 2)?)?;
+ catalog_b.register_schema("schema_b", Arc::new(schema_b));
+ ctx.register_catalog("catalog_b", Arc::new(catalog_b));
+
+ let result = plan_and_collect(
+ &mut ctx,
+ "SELECT cat, SUM(i) AS total FROM (
+ SELECT i, 'a' AS cat FROM catalog_a.schema_a.table_a
+ UNION ALL
Review comment:
@Dandandan `UNION` is already being used!
##########
File path: rust/benchmarks/src/bin/tpch.rs
##########
@@ -157,9 +157,9 @@ async fn benchmark(opt: BenchmarkOpt) -> Result<Vec<arrow::record_batch::RecordB
table,
start.elapsed().as_millis()
);
- ctx.register_table(table, Arc::new(memtable));
+ ctx.register_table(*table, Arc::new(memtable))?;
Review comment:
The change to make `register_table` fallible is a breaking change, but a very reasonable one I think.
##########
File path: rust/datafusion/src/catalog/schema.rs
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of schemas,
+//! representing collections of named tables.
+
+use crate::datasource::TableProvider;
+use crate::error::{DataFusionError, Result};
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a schema, comprising a number of named tables.
+pub trait SchemaProvider: Sync + Send {
+ /// Returns the schema provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available table names in this schema.
+ fn table_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific table from the schema by name, provided it exists.
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>>;
+
+ /// If supported by the implementation, adds a new table to this schema.
+ /// If a table of the same name existed before, it is replaced in the schema and returned.
+ #[allow(unused_variables)]
+ fn register_table(
+ &self,
+ name: String,
+ table: Arc<dyn TableProvider>,
Review comment:
```suggestion
fn register_table(
&self,
_name: String,
_table: Arc<dyn TableProvider>,
```
##########
File path: rust/benchmarks/src/bin/tpch.rs
##########
@@ -1105,7 +1105,7 @@ fn get_table(
table: &str,
table_format: &str,
max_concurrency: usize,
-) -> Result<Arc<dyn TableProvider + Send + Sync>> {
Review comment:
FWIW I was confused about this change initially until I saw that you added `Send + Sync` to the `TableProvider` trait itself. I think that is a good change (so we don't have to remember to add `Send + Sync` everywhere)
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -564,13 +634,30 @@ impl ExecutionConfig {
self.optimizers.push(optimizer_rule);
self
}
+
+ /// Selects a name for the default catalog and schema
+ pub fn with_default_catalog_and_schema(
Review comment:
I like this API. 👍
I wonder if we need an API for getting `default_catalog` and `default_schema`?
##########
File path: rust/datafusion/src/catalog/schema.rs
##########
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Describes the interface and built-in implementations of schemas,
+//! representing collections of named tables.
+
+use crate::datasource::TableProvider;
+use crate::error::{DataFusionError, Result};
+use std::any::Any;
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+/// Represents a schema, comprising a number of named tables.
+pub trait SchemaProvider: Sync + Send {
+ /// Returns the schema provider as [`Any`](std::any::Any)
+ /// so that it can be downcast to a specific implementation.
+ fn as_any(&self) -> &dyn Any;
+
+ /// Retrieves the list of available table names in this schema.
+ fn table_names(&self) -> Vec<String>;
+
+ /// Retrieves a specific table from the schema by name, provided it exists.
+ fn table(&self, name: &str) -> Option<Arc<dyn TableProvider>>;
+
+ /// If supported by the implementation, adds a new table to this schema.
+ /// If a table of the same name existed before, it is replaced in the schema and returned.
+ #[allow(unused_variables)]
+ fn register_table(
+ &self,
+ name: String,
+ table: Arc<dyn TableProvider>,
Review comment:
I think that pattern is more "idiomatic rust" but the unused variables thing works too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org