You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/05/08 14:01:16 UTC

[I] Support "User defined coercsion" function [datafusion]

alamb opened a new issue, #10423:
URL: https://github.com/apache/datafusion/issues/10423

   ### Is your feature request related to a problem or challenge?
   
   DataFusion automatically "coerces" (see [docs here](https://docs.rs/datafusion/latest/datafusion/logical_expr/type_coercion/index.html)) input argument types to match the types required of operations or functions. 
   
   For functions, this is described by a desired [TypeSignature](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TypeSignature.html#)
   ```rust
   pub enum TypeSignature {
       Variadic(Vec<DataType>),
       VariadicEqual,
       VariadicAny,
       Uniform(usize, Vec<DataType>),
       Exact(Vec<DataType>),
       Any(usize),
       OneOf(Vec<TypeSignature>),
       ArraySignature(ArrayFunctionSignature),
   }
   ```
   
   However, some functions have special hard coded coercion logic such as sum and count (TODO link) as well as some Array functions like `make_array`. We started down the path of encoding the special array semantics into `TypeSignature` (see `ArrayFunctionSignature`))
   
   However, as we continue to find other examples of different desired rules (most recently in sum and count), `TypeSignature` will grow and become more and more specialized
   
   
   
   ### Describe the solution you'd like
   
   @jayzhan211  had a great suggestion https://github.com/apache/datafusion/pull/10268#discussion_r1593511241 that in addition to encoding common coercion behaviors in `TypeSignature`, we can also add a variant of `TypeSignature` that permits user defined coercion rules
   
   
   
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support "User defined coercion" rules [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #10423:
URL: https://github.com/apache/datafusion/issues/10423#issuecomment-2101702285

   I found that to have `coerce_types` for both scalarUDF and aggregateUDF, I would need to introduce a more general trait for both function
   
   ```rust
   impl UDFImpl for T {
       fn name(&self) -> &str {
          &self.name
       }
   
       fn coerce_types(&self, data_types: &[DataType]) -> Result<Vec<DataType>> {
           not_impl_err!("Function {} does not implement coerce_types", self.name)
       }
   }
   
   impl ScalarUDFImpl: UDFImpl for T
   impl AggregateUDFImpl: UDFImpl for T
   ```
   Then we can have
   ```
   fn coerce_arguments_for_signature(
       expressions: Vec<Expr>,
       schema: &DFSchema,
       signature: &Signature,
       func: Arc<dyn UDFImpl>,
   ) -> Result<Vec<Expr>> {}
   ```
   
   Another alternative is having a duplicate function for scalar and aggregate for a related function
   
   ```
   fn coerce_arguments_for_signature(
       expressions: Vec<Expr>,
       schema: &DFSchema,
       signature: &Signature,
       func: &ScalarUDF,
   ) -> Result<Vec<Expr>> {}
   
   fn coerce_arguments_for_signature(
       expressions: Vec<Expr>,
       schema: &DFSchema,
       signature: &Signature,
       func: &AggregateUDF,
   ) -> Result<Vec<Expr>> {}
   ```
   
   I think the first option is potentially beneficial in the long run(?)
   @alamb what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support "User defined coercion" rules [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #10423: Support "User defined coercion" rules
URL: https://github.com/apache/datafusion/issues/10423


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support "User defined coercion" rules [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10423:
URL: https://github.com/apache/datafusion/issues/10423#issuecomment-2102415148

   It is a good observation that  `ScalarUDFImpl` and `AggregateUDFImpl` don't share a common base trait  and thus adding functionality that affects both requires duplication of code
   
   > I would need to introduce a more general trait for both function
   > Another alternative is having a duplicate function for scalar and aggregate for a related function
   
   I agree with your analysis of the tradeoffs: a common base trait would result in less duplication in DataFusion
   
   However, I personally prefer duplicating `coerce_arguments_for_signature`  in each trait rather than introducing a common base trait because:
   1. It is backwards compatible (not an API change for the existing library of functions)
   2. Makes it slightly easier to implement ScalarUDF and AggregateUDF (especially when new to rust) -- rather than two `impl`s for your function, you only need one
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org