You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/05/08 14:01:16 UTC
[I] Support "User defined coercsion" function [datafusion]
alamb opened a new issue, #10423:
URL: https://github.com/apache/datafusion/issues/10423
### Is your feature request related to a problem or challenge?
DataFusion automatically "coerces" (see [docs here](https://docs.rs/datafusion/latest/datafusion/logical_expr/type_coercion/index.html)) input argument types to match the types required of operations or functions.
For functions, this is described by a desired [TypeSignature](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TypeSignature.html#)
```rust
pub enum TypeSignature {
Variadic(Vec<DataType>),
VariadicEqual,
VariadicAny,
Uniform(usize, Vec<DataType>),
Exact(Vec<DataType>),
Any(usize),
OneOf(Vec<TypeSignature>),
ArraySignature(ArrayFunctionSignature),
}
```
However, some functions have special hard coded coercion logic such as sum and count (TODO link) as well as some Array functions like `make_array`. We started down the path of encoding the special array semantics into `TypeSignature` (see `ArrayFunctionSignature`))
However, as we continue to find other examples of different desired rules (most recently in sum and count), `TypeSignature` will grow and become more and more specialized
### Describe the solution you'd like
@jayzhan211 had a great suggestion https://github.com/apache/datafusion/pull/10268#discussion_r1593511241 that in addition to encoding common coercion behaviors in `TypeSignature`, we can also add a variant of `TypeSignature` that permits user defined coercion rules
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support "User defined coercion" rules [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #10423:
URL: https://github.com/apache/datafusion/issues/10423#issuecomment-2101702285
I found that to have `coerce_types` for both scalarUDF and aggregateUDF, I would need to introduce a more general trait for both function
```rust
impl UDFImpl for T {
fn name(&self) -> &str {
&self.name
}
fn coerce_types(&self, data_types: &[DataType]) -> Result<Vec<DataType>> {
not_impl_err!("Function {} does not implement coerce_types", self.name)
}
}
impl ScalarUDFImpl: UDFImpl for T
impl AggregateUDFImpl: UDFImpl for T
```
Then we can have
```
fn coerce_arguments_for_signature(
expressions: Vec<Expr>,
schema: &DFSchema,
signature: &Signature,
func: Arc<dyn UDFImpl>,
) -> Result<Vec<Expr>> {}
```
Another alternative is having a duplicate function for scalar and aggregate for a related function
```
fn coerce_arguments_for_signature(
expressions: Vec<Expr>,
schema: &DFSchema,
signature: &Signature,
func: &ScalarUDF,
) -> Result<Vec<Expr>> {}
fn coerce_arguments_for_signature(
expressions: Vec<Expr>,
schema: &DFSchema,
signature: &Signature,
func: &AggregateUDF,
) -> Result<Vec<Expr>> {}
```
I think the first option is potentially beneficial in the long run(?)
@alamb what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support "User defined coercion" rules [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #10423: Support "User defined coercion" rules
URL: https://github.com/apache/datafusion/issues/10423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support "User defined coercion" rules [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10423:
URL: https://github.com/apache/datafusion/issues/10423#issuecomment-2102415148
It is a good observation that `ScalarUDFImpl` and `AggregateUDFImpl` don't share a common base trait and thus adding functionality that affects both requires duplication of code
> I would need to introduce a more general trait for both function
> Another alternative is having a duplicate function for scalar and aggregate for a related function
I agree with your analysis of the tradeoffs: a common base trait would result in less duplication in DataFusion
However, I personally prefer duplicating `coerce_arguments_for_signature` in each trait rather than introducing a common base trait because:
1. It is backwards compatible (not an API change for the existing library of functions)
2. Makes it slightly easier to implement ScalarUDF and AggregateUDF (especially when new to rust) -- rather than two `impl`s for your function, you only need one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org