You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "goldmedal (via GitHub)" <gi...@apache.org> on 2025/03/23 07:38:35 UTC

[I] Support Custom Function Registration with Catalog and Schema [datafusion]

goldmedal opened a new issue, #15363:
URL: https://github.com/apache/datafusion/issues/15363

   ### Is your feature request related to a problem or challenge?
   
   In my case, I would like to extend a default function's supporting signature. For example, the `avg` function in DataFusion doesn't support calculating the Duration average. 
   However, suppose I want to register a custom `avg` for Duration. In that case, it means that I need to re-implement the whole `avg` function because the new implementation will override the previous one when registered. I want to propose an approach to register the custom implementation and keep the original implementation at the same time.
   
   ### Describe the solution you'd like
   
   In Postgres or DuckDB, their function definitions have catalog and schema. We can create a macro or UDF with the specific catalog or schema prefix. Then, we can set up the search_path to decide where to find the functions. If no matching candidates are found in the search path, we can find the candidate in the default path.
   
   If we have similar features, I imagine I can follow these steps to extend an existing function easily:
   
   1. Create a schema `self`.
   2. Implement an `avg` function for Duration.
   3. Register the `self.avg` function.
   4. Set `search_path` to `self`.
   5. Query the SQL `select avg(time_col1 - time_col2) from table1` (Invoke self.avg internally).
   6. Query the SQL `select avg(int_col) from table1` (Invoke default avg internally).
   
   To approach this proposal, I think we need two new features:
   
   - Register and use UDFs with the catalog and schema. I think we can do something at:
       https://github.com/apache/datafusion/blob/d460abb63f6ff3abce6de022834f6f17c77db411/datafusion/core/src/execution/session_state.rs#L1773-L1778
   - Implement `search_path` for searching function candidates.
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support Custom Function Registration with Catalog and Schema [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15363:
URL: https://github.com/apache/datafusion/issues/15363#issuecomment-2758213146

   BTW @svranesevic  just added support for Min/Max for Duration (we can probably implement the equivalent for average). 
   - https://github.com/apache/datafusion/pull/15310
   
   I will file a ticket


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support Custom Function Registration with Catalog and Schema [datafusion]

Posted by "leoyvens (via GitHub)" <gi...@apache.org>.
leoyvens commented on issue #15363:
URL: https://github.com/apache/datafusion/issues/15363#issuecomment-2906881168

   A backwards compatibility issue to consider is that `.` is currently valid in udf identifiers. I currently abuse this to emulate schema namespacing for UDFs even though DataFusion doesn't support it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support Custom Function Registration with Catalog and Schema [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15363:
URL: https://github.com/apache/datafusion/issues/15363#issuecomment-2758245006

   - Filed  https://github.com/apache/datafusion/issues/15458 to support Duration in average


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support Custom Function Registration with Catalog and Schema [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15363:
URL: https://github.com/apache/datafusion/issues/15363#issuecomment-2758210504

   In general, adding Catalog/Schema information to functions makes sense to me, but I think it would be a non trivial effort
   
   > However, suppose I want to register a custom avg for Duration. In that case, it means that I need to re-implement the whole avg function because the new implementation will override the previous one when registered. 
   
   Another approach could be to implement your own `avg` function as a wrapper around `avg` that called into the provided implementation. For example
   ```rust
   /// An average that also supports duration
   pub struct AvgWithDuration {
     // standard DataFusion average impmenetation
     inner: Average
   }
   
   /// implement aggregate udf for new average, call
   impl AggregateUDFImpl for AvgWithDuration {
   ...
         fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
          // handle duration
          if arg_types[0] == DataType::Duration {
            Ok(DataType::Duration)
          }
           // otherwise, fall back to inner implementation
           self.inner.return_type(arg_types)
       }
   
       fn accumulator(&self, acc_args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
          // handle duration, with accumulator implemented for Duration
          if arg_types[0] == DataType::Duration {
            Ok(Box::new(DurationAccumulator(...))
          }
           // otherwise, fall back to inner implementation
           self.inner.return_type(arg_types)
       }
   ...
   }
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org