You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nuno-faria (via GitHub)" <gi...@apache.org> on 2025/11/19 12:29:17 UTC
[I] Enabling `expand_views_at_output` config changes column names [datafusion]
nuno-faria opened a new issue, #18818:
URL: https://github.com/apache/datafusion/issues/18818
### Describe the bug
When enabling the `expand_views_at_output` config to convert `UTF8View` to `UTF8Large`, the names of the converted columns change, being prefixed with the relation name. I think the cause is that a `CAST` is added to change the type, meaning `Expr::qualified_name` will return "table.column" instead of just column:
```rust
// when we have a CAST we end up at the last match arm
pub fn qualified_name(&self) -> (Option<TableReference>, String) {
match self {
Expr::Column(Column {
relation,
name,
spans: _,
}) => (relation.clone(), name.clone()),
Expr::Alias(Alias { relation, name, .. }) => (relation.clone(), name.clone()),
_ => (None, self.schema_name().to_string()),
}
}
// which in turn calls
SchemaDisplay(self)
// which for cast simply calls SchemaDisplay(self) of the inner expression
Expr::Cast(Cast { expr, .. }) | Expr::TryCast(TryCast { expr, .. }) => {
write!(f, "{}", SchemaDisplay(expr))
}
// which for Column calls
impl fmt::Display for Column {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "{}", self.flat_name())
}
}
// which includes the relation + name, unlike the original qualified_name for a regular Column
```
I think one approach would be to update `qualified_name` and adding a match for casts. I would be happy to fix this, if it is indeed a bug a not expected behavior.
### To Reproduce
```rust
use datafusion::error::Result;
use datafusion::prelude::{ParquetReadOptions, SessionContext};
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
ctx.sql("copy (select 1 as k, 'a' as v) to 't.parquet'")
.await?
.collect()
.await?;
ctx.register_parquet("t", "t.parquet", ParquetReadOptions::new())
.await?;
let df = ctx.sql("select * from t").await?;
df.clone().show().await?;
println!("{:?}", df.collect().await?[0].schema());
ctx.sql("set datafusion.optimizer.expand_views_at_output = true")
.await?
.collect()
.await?;
let df = ctx.sql("select * from t").await?;
df.clone().show().await?;
println!("{:?}", df.collect().await?[0].schema());
Ok(())
}
```
`k` remains the same but `v` changes:
```
+---+---+
| k | v |
+---+---+
| 1 | a |
+---+---+
Schema { fields: [Field { name: "k", data_type: Int64 }, Field { name: "v", data_type: Utf8View }], metadata: {} }
+---+-----+
| k | t.v |
+---+-----+
| 1 | a |
+---+-----+
Schema { fields: [Field { name: "k", data_type: Int64 }, Field { name: "t.v", data_type: LargeUtf8 }], metadata: {} }
```
### Expected behavior
Maintaining the original column names.
### Additional context
Tested on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Enabling `expand_views_at_output` config changes column names [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #18818: Enabling `expand_views_at_output` config changes column names
URL: https://github.com/apache/datafusion/issues/18818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org