You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "dadepo (via GitHub)" <gi...@apache.org> on 2023/06/05 07:26:53 UTC
[GitHub] [arrow-datafusion] dadepo opened a new issue, #6542: Error: push_down_projection
dadepo opened a new issue, #6542:
URL: https://github.com/apache/arrow-datafusion/issues/6542
### Describe the bug
I have a query that has the following structure
```
let df = ctx.sql(r#"
with input as (SELECT * FROM demo OFFSET 1 limit 3)
SELECT cols_no_available
from input
cross join (select payload from demo limit 1) cross_table
"#).await?;
```
`cols_no_available` does not exist and was purposely included in the example to show the error message, which is
```
Error: Schema error: No field named cols_no_available. Valid fields are input.row_input, cross_table.payload.
Caused by:
No field named cols_no_available. Valid fields are input.row_input, cross_table.payload.
```
It clearly says `cols_no_available` column is not available, but that `input.row_input`, and `cross_table.payload` are available. But when I run this query, where I want to select cross_table.payload
```
let df = ctx.sql(r#"
with input as (SELECT * FROM demo OFFSET 1 limit 3)
SELECT cross_table.payload
from input
cross join (select payload from demo limit 1) cross_table
"#).await?;
```
I get the error
```
Error: push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "cross_table" }), field: Field { name: "payload", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "input" }), field: Field { name: "id", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }, DFField { qualifier: Some(Bare { table: "cross_table" }), field: Field { name: "payload", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
on the other hand, the query runs fine, having `input.row_input` instead of `cross_table.payload`
### To Reproduce
Construct a query where a CTE is being used with a cross join. The cross join cannot be accessed.
### Expected behavior
It should be able to query the columns in a cross join with a CTE
### Additional context
```
datafusion = { version = "24.0.0" }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on issue #6542: Error: push_down_projection
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on issue #6542:
URL: https://github.com/apache/arrow-datafusion/issues/6542#issuecomment-1577043692
i did some test on version 25.0.0, it's look
```
#[tokio::test]
async fn test_6542() -> Result<()> {
let ctx = create_join_context()?;
ctx.sql(
r#"
with input as (SELECT * FROM demo OFFSET 1 limit 3)
SELECT cross_table.payload
from input
cross join (select payload from demo limit 1) cross_table
"#,
)
.await?
.explain(true, false)?
.show()
.await?;
Ok(())
}
```
if you still got error, please upload output of explain. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6542: Error: push_down_projection
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6542:
URL: https://github.com/apache/arrow-datafusion/issues/6542#issuecomment-1666541596
Closing this one down -- please reopen or open a new ticket if you see this symptom again.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on issue #6542: Error: push_down_projection
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on issue #6542:
URL: https://github.com/apache/arrow-datafusion/issues/6542#issuecomment-1577030335
i did some test on version 25.0.0. it's worked.
```
#[tokio::test]
async fn test_6542() -> Result<()> {
let ctx = create_join_context()?;
let sql_results = ctx
.sql(
"with input as (SELECT * FROM t1 OFFSET 1 limit 3)
SELECT a
from input
cross join (select b from t2 limit 1) cross_table",
)
.await?
.explain(true, false)?
.show()
.await?;
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #6542: Error: push_down_projection
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6542: Error: push_down_projection
URL: https://github.com/apache/arrow-datafusion/issues/6542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org