You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/29 22:40:54 UTC
[GitHub] [arrow-datafusion] dmitrijoseph opened a new issue, #4428: with_column_renamed is always a NO OP when table name is ?table?
dmitrijoseph opened a new issue, #4428:
URL: https://github.com/apache/arrow-datafusion/issues/4428
```
let test_df = ctx.read_csv("test.csv", CsvReadOptions::new()).await?;
let test_df = test_df.with_column_renamed("id", "renamedID")?;
println!("{:#?}", test_df.explain(true, true)?);
```
```
+---------------+-----------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+-----------------------------------------------------------------------------------------------------------------+
| logical_plan | TableScan: ?table? projection=[id, name] |
| physical_plan | CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
| | |
+---------------+-----------------------------------------------------------------------------------------------------------------+
```
This time using `?table?` as a qualifier
```
let test_df = ctx.read_csv("test.csv", CsvReadOptions::new()).await?;
let test_df = test_df.with_column_renamed("?table?.id", "renamedID")?;
test_df.explain(false, false)?.show().await?;
```
```
Error: SchemaError(FieldNotFound { field: Column { relation: None, name: "?table?.id" }, valid_fields: Some([Column { relation: Some("?table?"), name: "id" }, Column { relation: Some("?table?"), name: "name" }]) })
```
This time registering table so it has a set table name
```
let test_df = ctx.read_csv("test.csv", CsvReadOptions::new()).await?;
ctx.register_table("t1", test_df)?;
let test_df = ctx.table("t1")?;
let test_df = test_df.with_column_renamed("id", "renamedID")?;
test_df.explain(false, false)?.show().await?;
```
```
+---------------+-------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+-------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: ?table?.id, ?table?.name, alias=t1 |
| | TableScan: ?table? projection=[id, name] |
| physical_plan | ProjectionExec: expr=[id@0 as id, name@1 as name] |
| | CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
| | |
+---------------+-------------------------------------------------------------------------------------------------------------------+
```
This time registering table so it has a set table name and using that table name as a qualifier
```
let test_df = ctx.read_csv("test.csv", CsvReadOptions::new()).await?;
ctx.register_table("t1", test_df)?;
let test_df = ctx.table("t1")?;
let test_df = test_df.with_column_renamed("t1.id", "renamedID")?;
test_df.explain(false, false)?.show().await?;
```
```
+---------------+---------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+---------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: t1.id AS renamedId, t1.name |
| | Projection: ?table?.id, ?table?.name, alias=t1 |
| | TableScan: ?table? projection=[id, name] |
| physical_plan | ProjectionExec: expr=[id@0 as renamedId, name@1 as name] |
| | ProjectionExec: expr=[id@0 as id, name@1 as name] |
| | CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
| | |
+---------------+---------------------------------------------------------------------------------------------------------------------+
```
I have two lines of thinking on this
1. `with_column_renamed` seems to be the only function in the DataFrame API that always requires the use of the fully qualified column name, this should be changed to use the unqualified column name.
2. Why are tables being set to `?table?` ? I assume this was chosen because it isn't valid sql but why not always generate a valid table name from functions like `ctx.read_csv()`. The current implementation is awkward because using `?table?` as a qualifier returns an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #4428: with_column_renamed is always a NO OP when table name is ?table?
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #4428: with_column_renamed is always a NO OP when table name is ?table?
URL: https://github.com/apache/arrow-datafusion/issues/4428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jefffrey commented on issue #4428: with_column_renamed is always a NO OP when table name is ?table?
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #4428:
URL: https://github.com/apache/arrow-datafusion/issues/4428#issuecomment-1426703229
I agree with point 1, in that it should be able to infer the full qualified column from an unqualified column name, as other methods in the Dataframe API are able to, though will need to enforce ambiguity checks to catch any ambiguous references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org