You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/31 20:44:02 UTC
[GitHub] [arrow] alamb opened a new pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
alamb opened a new pull request #9866:
URL: https://github.com/apache/arrow/pull/9866
(built on https://github.com/apache/arrow/pull/9840, so draft until that is merged)
# Rationale
Accessing the list of columns via `select * from information_schema.columns` (introduced in https://github.com/apache/arrow/pull/9840) is a lot to type
See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#
# Proposal
Add support for `SHOW COLUMNS FROM <table>` command.
Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html
# Example Use
Setup:
```
echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli
```
Then run :
```
> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
> 0 rows in set. Query took 0 seconds.
show columns from t;
show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.
```
Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution). Ideally this should also work:
```
> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")
```
# Commentary
This is the sister PR to `SHOW TABLES` here: https://github.com/apache/arrow/pull/9847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#issuecomment-813562273
Thanks. I made a PR with some proposed docs here: https://github.com/apache/arrow/pull/9895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#issuecomment-813373874
FYI @returnString and @seddonm1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#discussion_r605964671
##########
File path: rust/datafusion/src/sql/planner.rs
##########
@@ -1305,6 +1304,74 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
)))
}
}
+
+ fn show_columns_to_plan(
+ &self,
+ extended: bool,
+ full: bool,
+ table_name: &ObjectName,
+ filter: Option<&ShowStatementFilter>,
+ ) -> Result<LogicalPlan> {
+ if filter.is_some() {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS with WHERE or LIKE is not supported".to_string(),
+ ));
+ }
+
+ if !self.has_table("information_schema", "columns") {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS is not supported unless information_schema is enabled"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .schema_provider
+ .get_table_provider(table_name.try_into()?)
+ .is_none()
+ {
+ return Err(DataFusionError::Plan(format!(
+ "Unknown relation for SHOW COLUMNS: {}",
+ table_name
+ )));
+ }
+
+ // Figure out the where clause
+ let columns = vec!["table_name", "table_schema", "table_catalog"].into_iter();
+ let where_clause = table_name
+ .0
+ .iter()
+ .rev()
Review comment:
Why `rev` and not listing the items in column in reverse?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#discussion_r607075410
##########
File path: rust/datafusion/src/sql/planner.rs
##########
@@ -1305,6 +1304,74 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
)))
}
}
+
+ fn show_columns_to_plan(
+ &self,
+ extended: bool,
+ full: bool,
+ table_name: &ObjectName,
+ filter: Option<&ShowStatementFilter>,
+ ) -> Result<LogicalPlan> {
+ if filter.is_some() {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS with WHERE or LIKE is not supported".to_string(),
+ ));
+ }
+
+ if !self.has_table("information_schema", "columns") {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS is not supported unless information_schema is enabled"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .schema_provider
+ .get_table_provider(table_name.try_into()?)
+ .is_none()
+ {
+ return Err(DataFusionError::Plan(format!(
+ "Unknown relation for SHOW COLUMNS: {}",
+ table_name
+ )));
+ }
+
+ // Figure out the where clause
+ let columns = vec!["table_name", "table_schema", "table_catalog"].into_iter();
+ let where_clause = table_name
+ .0
+ .iter()
+ .rev()
Review comment:
You are totally right 👍 I don't have a suggestion for an easier way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#issuecomment-812192692
Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9866:
URL: https://github.com/apache/arrow/pull/9866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#issuecomment-811575690
https://issues.apache.org/jira/browse/ARROW-12109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#discussion_r605964671
##########
File path: rust/datafusion/src/sql/planner.rs
##########
@@ -1305,6 +1304,74 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
)))
}
}
+
+ fn show_columns_to_plan(
+ &self,
+ extended: bool,
+ full: bool,
+ table_name: &ObjectName,
+ filter: Option<&ShowStatementFilter>,
+ ) -> Result<LogicalPlan> {
+ if filter.is_some() {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS with WHERE or LIKE is not supported".to_string(),
+ ));
+ }
+
+ if !self.has_table("information_schema", "columns") {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS is not supported unless information_schema is enabled"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .schema_provider
+ .get_table_provider(table_name.try_into()?)
+ .is_none()
+ {
+ return Err(DataFusionError::Plan(format!(
+ "Unknown relation for SHOW COLUMNS: {}",
+ table_name
+ )));
+ }
+
+ // Figure out the where clause
+ let columns = vec!["table_name", "table_schema", "table_catalog"].into_iter();
+ let where_clause = table_name
+ .0
+ .iter()
+ .rev()
Review comment:
Why `rev` and not listing the items in `columns` in reverse?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#discussion_r605970581
##########
File path: rust/datafusion/src/sql/planner.rs
##########
@@ -1305,6 +1304,74 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
)))
}
}
+
+ fn show_columns_to_plan(
+ &self,
+ extended: bool,
+ full: bool,
+ table_name: &ObjectName,
+ filter: Option<&ShowStatementFilter>,
+ ) -> Result<LogicalPlan> {
+ if filter.is_some() {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS with WHERE or LIKE is not supported".to_string(),
+ ));
+ }
+
+ if !self.has_table("information_schema", "columns") {
+ return Err(DataFusionError::Plan(
+ "SHOW COLUMNS is not supported unless information_schema is enabled"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .schema_provider
+ .get_table_provider(table_name.try_into()?)
+ .is_none()
+ {
+ return Err(DataFusionError::Plan(format!(
+ "Unknown relation for SHOW COLUMNS: {}",
+ table_name
+ )));
+ }
+
+ // Figure out the where clause
+ let columns = vec!["table_name", "table_schema", "table_catalog"].into_iter();
+ let where_clause = table_name
+ .0
+ .iter()
+ .rev()
Review comment:
I was thinking because the identifier may not have a `table_catalog` or `table_schema`
So you have to handle the case of
`table_name` (position 0)
`table_name` (position 1), `schema_name` (postition 0)
`table_name` (postition 2), `schema_name` (postition 1), `catalog_name` (position 0),
So this formulation was what I could come up with that would match them up to the information_schema column names
I am open to other ways of doing it as well if you have suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9866: ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9866:
URL: https://github.com/apache/arrow/pull/9866#issuecomment-813671922
I plan to wait until tomorrow to merge this in case @andygrove has any comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org