You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/09/04 21:21:08 UTC
[I] Extension metadata dropped from literals in SQL VALUES clause [datafusion]
paleolimbot opened a new issue, #17425:
URL: https://github.com/apache/datafusion/issues/17425
### Describe the bug
Function calls that return scalars can be used in SQL VALUES; however if they contain extension metadata the metadata is dropped.
### To Reproduce
Output:
```
Regular select:
Field { name: "extension", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"ARROW:extension:metadata": "foofy.foofy"} }
VALUES select:
Field { name: "extension", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
```
```rust
use std::collections::HashMap;
use datafusion::{
arrow::datatypes::DataType,
logical_expr::{ScalarUDFImpl, Signature, Volatility},
prelude::*,
};
#[tokio::main]
async fn main() {
let ctx = SessionContext::new();
ctx.register_udf(MakeExtension::default().into());
let batches = ctx
.sql("SELECT make_extension('foofy zero') as extension")
.await
.unwrap()
.collect()
.await
.unwrap();
println!("Regular select:");
println!("{:?}", batches[0].schema().field(0));
let batches = ctx
.sql(
"
SELECT extension FROM (VALUES
('one', make_extension('foofy one')),
('two', make_extension('foofy two')),
('three', make_extension('foofy three')))
AS t(string, extension)
",
)
.await
.unwrap()
.collect()
.await
.unwrap();
println!("\nVALUES select:");
println!("{:?}", batches[0].schema().field(0));
}
#[derive(Debug)]
struct MakeExtension {
signature: Signature,
}
impl Default for MakeExtension {
fn default() -> Self {
Self {
signature: Signature::user_defined(Volatility::Immutable),
}
}
}
impl ScalarUDFImpl for MakeExtension {
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn name(&self) -> &str {
"make_extension"
}
fn signature(&self) -> &Signature {
&self.signature
}
fn coerce_types(&self, arg_types: &[DataType]) -> datafusion::error::Result<Vec<DataType>> {
Ok(arg_types.to_vec())
}
fn return_type(&self, _arg_types: &[DataType]) -> datafusion::error::Result<DataType> {
unreachable!("This shouldn't have been called")
}
fn return_field_from_args(
&self,
args: datafusion::logical_expr::ReturnFieldArgs,
) -> datafusion::error::Result<datafusion::arrow::datatypes::FieldRef> {
Ok(args.arg_fields[0]
.as_ref()
.clone()
.with_metadata(HashMap::from([(
"ARROW:extension:metadata".to_string(),
"foofy.foofy".to_string(),
)]))
.into())
}
fn invoke_with_args(
&self,
args: datafusion::logical_expr::ScalarFunctionArgs,
) -> datafusion::error::Result<datafusion::logical_expr::ColumnarValue> {
Ok(args.args[0].clone())
}
}
```
### Expected behavior
I would have expected the field metadata (if identical for all items) to be propagated to the schema of the values expression. This does bring the complexity of type equality, but byte-for-byte hash map equality should be safe. A "user defined extension type" (if there ever is one) could define a more lenient equality checker (e.g., JSON object metadata equality for extension types whose serialization is JSON).
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extension metadata dropped from literals in SQL VALUES clause [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #17425:
URL: https://github.com/apache/datafusion/issues/17425#issuecomment-3261561209
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extension metadata dropped from literals in SQL VALUES clause [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17425: Extension metadata dropped from literals in SQL VALUES clause
URL: https://github.com/apache/datafusion/issues/17425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extension metadata dropped from literals in SQL VALUES clause [datafusion]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #17425:
URL: https://github.com/apache/datafusion/issues/17425#issuecomment-3274873747
I took some time investigating this. Schema of VALUES was dropped when constructing the `LogicalPlan::Values` logical query plan:
https://github.com/apache/datafusion/blob/5bb476f014f064c9b7090b9fa0f0991777563b62/datafusion/expr/src/logical_plan/builder.rs#L301-L332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extension metadata dropped from literals in SQL VALUES clause [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #17425:
URL: https://github.com/apache/datafusion/issues/17425#issuecomment-3280609308
> I took some time investigating this. Schema of VALUES was dropped when constructing the `LogicalPlan::Values` logical query plan:
@Kontinuation if you already have solution. feel free to take these issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org