You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "colinmarc (via GitHub)" <gi...@apache.org> on 2025/11/25 14:46:27 UTC
[I] Implementing `CatalogProvider` breaks quoted CTE names [datafusion]
colinmarc opened a new issue, #18932:
URL: https://github.com/apache/datafusion/issues/18932
### Describe the bug
We use a custom `CatalogProvider`. We've discovered an edge case where a quoted CTE name breaks using a custom catalog provider, but not when using `MemoryCatalogProvider`. In other words, a query like:
```
with foobar as (select count(*) from orders) select * from "foobar"
```
Hits the catalog provider for `foobar`, even though it's a CTE table, and the catalog doesn't have any record of in it advance, obviously. Not quoting `"foobar"` or just using `ctx.register_table("orders", ...)` both cause the query to work.
### To Reproduce
Here is some code that reproduces:
```rust
#[derive(Debug)]
struct MyCatalog;
#[derive(Debug)]
struct MySchema(String);
impl CatalogProvider for MyCatalog {
fn as_any(&self) -> &dyn Any {
self
}
fn schema_names(&self) -> Vec<String> {
Vec::new()
}
fn schema(&self, name: &str) -> Option<Arc<dyn SchemaProvider>> {
Some(Arc::new(MySchema(name.to_owned())))
}
}
#[async_trait::async_trait]
impl SchemaProvider for MySchema {
fn as_any(&self) -> &dyn Any {
self
}
fn table_names(&self) -> Vec<String> {
todo!()
}
async fn table(
&self,
name: &str,
) -> Result<Option<Arc<dyn datafusion::catalog::TableProvider>>, DataFusionError> {
if name == "orders" {
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("name", DataType::Utf8, false),
]));
return Ok(Some(Arc::new(EmptyTable::new(schema))));
} else {
panic!("Table not found: {name}")
}
}
fn table_exist(&self, name: &str) -> bool {
name == "orders"
}
}
#[tokio::test]
async fn quoted_cte() -> anyhow::Result<()> {
let ctx = SessionContext::new();
ctx.register_catalog("datafusion", Arc::new(MyCatalog));
// If I comment out the line above and use this instead, both queries work.
//
// let schema = Arc::new(Schema::new(vec![
// Field::new("id", DataType::Int32, false),
// Field::new("name", DataType::Utf8, false),
// ]));
// let table = Arc::new(EmptyTable::new(schema));
// ctx.register_table("orders", table)?;
// NO QUOTING: This always works either way.
let res = ctx
.sql("with foobar as (select * from orders) select * from foobar")
.await?
.collect()
.await?;
println!("{}", pretty_format_batches(&res)?);
// QUOTING: This doesn't work with `MyCatalog`.
let res = ctx
.sql("with barbaz as (select * from orders) select * from \"barbaz\"")
.await?
.collect()
.await?;
println!("{}", pretty_format_batches(&res)?);
Ok(())
}
```
### Expected behavior
Both queries in the example should work with a custom catalog provider.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implementing `CatalogProvider` breaks quoted CTE names [datafusion]
Posted by "kysshsy (via GitHub)" <gi...@apache.org>.
kysshsy commented on issue #18932:
URL: https://github.com/apache/datafusion/issues/18932#issuecomment-3694599070
**Observation**
- In `statement_to_plan`, a query like `WITH barbaz AS (...) SELECT * FROM "barbaz"` is initially collected as a `TableReference` (looks like a regular table) during `resolve_table_references`.
- Because `visit_statement(statement, &mut visitor)` collects CTE refs and table refs *before* normalization, `barbaz` and `"barbaz"` are treated as different idents at that stage.
**However**
- During planning, `SqlToRel` re-resolves the reference: it first checks the CTE stack (`PlannerContext::get_cte`) and only falls back to the catalog/provider if the CTE isn’t found. In practice, the CTE is found there, so the query still succeeds.
**Uncertainty**
- It’s unclear whether emitting the quoted CTE name as a table ref in `resolve_table_references` is just a benign quirk or a bug (it triggers unnecessary catalog lookup and can fail with custom providers). I’m happy to submit a PR to align collection and CTE resolution so quoted CTE names aren’t emitted as table refs at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implementing `CatalogProvider` breaks quoted CTE names [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #18932: Implementing `CatalogProvider` breaks quoted CTE names
URL: https://github.com/apache/datafusion/issues/18932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implementing `CatalogProvider` breaks quoted CTE names [datafusion]
Posted by "colinmarc (via GitHub)" <gi...@apache.org>.
colinmarc commented on issue #18932:
URL: https://github.com/apache/datafusion/issues/18932#issuecomment-3576099259
This seems to be at least partially because in my example I'm `panic`ing in the `table` implementation (returning `Err` has the same effect). Returning `Ok(None)` does work, but I'll leave this open because I think it warrants further investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org