You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/02/20 07:49:02 UTC

[I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

alamb opened a new issue, #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285

   ### Is your feature request related to a problem or challenge?
   
   As part of making DataFusion even more customizable (https://github.com/apache/arrow-datafusion/issues/8045), it is valuable to let system designers mix and match different packages of functions to get the precise behavior they want (e.g. postgres style `to_date` or spark style `to_date`).
   
   To support this functionality as well as  to ensure the `ScalarUDF` API exposes the full power of DataFusion, we are in the process of extracting the "built in" functions out of the core and into separate crates.
   
   This epic tracks the work to actually move the functions out of the core datafusion crate (spread through `datafusion_expr` and `datafusion-physical-expr` and into the new `datafusion-functions` / `datafusion-functions-array` crates
   
   
   
   ### Describe the solution you'd like
   
   ## Tasks:
   
   Here is list of many of the items necessary to complete this transition.
   Eventually there should be tickets for all tasks, and there are tickets for some
   already, but I don't want to make 100s of tickets all at once. I plan to make
   more as we make it through more of this project.
   
   Anyone should feel free to make other tickets if they want to help with items below.
   
   # `math_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/math/mod.rs
   - [x] https://github.com/apache/arrow-datafusion/pull/9216
   - [ ] Port `abs` to datafusion_functions
   - [ ] Abs, Acos, Asin,
   - [ ] Atan, Atan2, Acosh, Asinh, Atanh,
   - [ ] Cbrt, Ceil, Cos, Cosh, Degrees, Exp, Factorial,
   - [ ] Floor, Gcd, Lcm, Ln, Log, Log10, Log2, Pi, Power,
   - [ ] Radians, Signum, Sin, Sinh, Sqrt,
   - [ ] Tan, Tanh, Trunc, Cot, Round, iszero
   
   ## `array_expressions`
   Note that given the size and specialization of these functions are put in their own subcrate
   [`datafusion-functions-array`](https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions-array)
   
   - [ ] ArrayToString (TODO find PR)
   - [ ] Move the `make_array` function into the `datafusion-array-expressions` crate -- TODO file and link ticket
   - [ ] ArrayAppend, ArraySort, ArrayConcat, ArrayHas, ArrayHasAll, ArrayHasAny,
   - [ ] ArrayPopFront, ArrayPopBack, ArrayDims, ArrayDistinct, ArrayElement,
   - [ ] ArrayEmpty, ArrayLength, ArrayNdims, ArrayPosition, ArrayPositions,
   - [ ] ArrayPrepend, ArrayRemove, ArrayRemoveN, ArrayRemoveAll, ArrayRepeat,
   - [ ] ArrayReplace, ArrayReplaceN, ArrayReplaceAll, ArraySlice,
   - [ ] ArrayIntersect, ArrayUnion, ArrayExcept,
   - [ ] Cardinality, ArrayResize, Flatten, Range, StringToArray,
   - [ ] `MakeArray`:  construct an array from columns (union/except depends on this)
   
   
   ## Core functions
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/core/mod.rs
   
   - [x] Create `core` module, extract `nullif`: https://github.com/apache/arrow-datafusion/pull/9216
   - [ ] Move `ArrowCast` to datafusion-functions
   - [ ] Move `ArrowTypeOf`: return the arrow type of a value
   - [ ] `Coalesce`: return the first non-null value
   - [ ] `Struct`: Create a struct
   - [ ] `NullIf`: return null if the two values are equal
   - [ ] `Random`: return a random number
   - [ ] `Nanvl`: return the first non-NaN value
   
   ## `crypto_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/crypto/mod.rs
   
   - [ ] Create `crypto` module in `datafusion/functions/src/crypto` and `crypto_expressions` feature flag, move `digest` function
   - [ ] Digest, MD5, SHA224, SHA256, SHA384, SHA512
   
   ## `string_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/string/mod.rs
   
   - [ ] Create `string` module in  `datafusion/functions/src/string` and `string_expressions` feature flag, move `ascii` function
   - [ ] ascii, bit_length, btrim, chr,
   - [ ] concat, concat_ws, ends_with, initcap,
   - [ ] instr, lower, ltrim, octet_length,
   - [ ] repeat, replace, rtrim, split_part,
   - [ ] starts_with, to_hex, trim, upper,
   - [ ] levenshtein, uuid, overlay
   
   ## `unicode_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs
   
   - [ ] Create `unicode` module in  `datafusion/functions/src/unicode` and `unicode_expressions` feature flag, move `charlength` function
   - [ ] CharLength,
   - [ ] Left, Lpad, Reverse, Right, Rpad,
   - [ ] Strpos, Substr,
   - [ ] Translate, SubstrIndex, FindInSet
   
   ## `regex_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/regexp/mod.rs
   
   - [ ] Create `regex` module in  `datafusion/functions/src/regex` and `regex_expressions` feature flag, move regexp_match
   - [ ] RegexpMatch, RegexpReplace
   - [ ] RegexpLike
   
   
   ## `datetime_expressions`
   These should be located in the `datafusion-functions` crate ([source link[(https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions))
   Code location: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/datetime/mod.rs
   
   - [ ] Create `datetime` module in `datafusion/functions/src/datetime` and `datetime_expressions` feature flag, move `date_part`
   - [ ] port benchmarks to datafusion-functions crate
   - [ ] date_part, date_trunc, date_bin,
   - [ ] to_timestamp, to_timestamp_millis, to_timestamp_micros, to_timestamp_nanos, to_timestamp_seconds,
   - [ ] from_unixtime, now, current_date, current_time
   
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   The organization was discussed in https://github.com/apache/arrow-datafusion/issues/9100
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1986677482

   Take crypto parts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2002909624

   It makes sense to me 👍
   
   On Mon, Mar 18, 2024, 11:53 AM Eren Avsarogullari ***@***.***>
   wrote:
   
   > @jayzhan211 <https://github.com/jayzhan211>, Just follow-up to our
   > previous discussion (single func per file), we almost complete
   > functions-array migration. Its kernels and udf source files include more
   > functions by our new pattern. Does it make sense to break them as well by
   > aligning the new pattern and consistency?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2002881891>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ADZCLR4R4QIYZRCVVY65K6LYYZQNNAVCNFSM6AAAAABDQUR3H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSHA4DCOBZGE>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2003035780

   @jayzhan211 Yes, makes sense. I have created #9660 for this refactoring. I will also include to be used `make_scalar_function` for `invoke` calls.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "PsiACE (via GitHub)" <gi...@apache.org>.
PsiACE commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987213882

   take string module  and ascii, bit_length, btrim, chr


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1957085509

   I'll try to do regex_expressions parts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1976668639

   While I worked on ArrayElement, I found that `GetFieldAccessExpr::ListIndex` uses `array_element` in physical-expr https://github.com/apache/arrow-datafusion/blob/581fd98270e69221689ddd4c37566ef620a67167/datafusion/physical-expr/src/expressions/get_indexed_field.rs#L252-L257 ListRange likewise, which is kernel function in `function-arrays` crate, instead of converting ArrayRef to Expr, I'm thinking of making array function kernel public. @alamb Do you think we can make those array functions public? And, is importing `function-arrays` to crate `physical-expr` a concern like crate `optimize` is?
   
   If neither is a good idea, we may need to rewrite ListIndex and ListRange to array function early.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1986710264

   Take ArrayEmpty and ArrayLength
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "yyy1000 (via GitHub)" <gi...@apache.org>.
yyy1000 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1962730263

   I filed #9336 which I think may be necessary before we port functions. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1993437980

   @jayzhan211 Thanks for notification. Yes, makes sense. We can define new `udf` definitions as separate file (single func per file). WDYT on `datafusion/functions-array/src/kernels.rs`? (Its size also increases)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9285: [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates
URL: https://github.com/apache/datafusion/issues/9285


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2001890115

   `ArrayExcept`: PR(#9634) has been submitted
   `ArrayRemove`, `ArrayRemoveN`, `ArrayRemoveAll`: Work in Progress(#9635)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1955596295

   Let me give this a try:
   
   >## `math_expressions`
   >
   >  [ ] Abs, Acos, Asin,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1974261686

   Take ArrayDims, ArrayNdims, Cardinality


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1966362401

   work on `array_concat`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1998693767

   Take `pop_front` and `pop_back`, they need to be moved together with slice


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1993280827

   @erenavsarogullari I think it is fine to split function to other file to avoid crazy conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2008509428

   Take lower, ltrim, octet_length


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1968089467

   Take Atan, Atan2, Acosh.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2002292512

   take array_remove family


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2001545142

   Take `ArrayReverse`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987270872

   take concat, concat_ws, ends_with, initcap,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "guojidan (via GitHub)" <gi...@apache.org>.
guojidan commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1961149680

   I'll try to do array_expressions parts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1977621477

   > I also encountered function [references ](https://github.com/apache/arrow-datafusion/blob/a84e5f89bd52d59c78f11fffbab89ed1d418538f/datafusion/sql/src/expr/mod.rs#L173)in the sql crate that I really can't see a way to work around unless we somehow allow SQLExpr to be a part of a ScalarUdf or similar.
   
   I was thinking in the sql parser we can just switch to lookung up functions by name (and the sql planner already has a function registry)
   
   > Hmm. I had the same issue with datebin being referenced in physical-expr 
   
   Perhaps it is time to make a new crate that contains common function implementation code 🤔 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2016587802

   take levenshtein, uuid, overlay


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "guojidan (via GitHub)" <gi...@apache.org>.
guojidan commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1966386379

   > work on `array_concat`
   
   hi @jayzhan211 I already work on `array_concat` #9343 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1966398787

   Take ArrayHas, ArrayHasAll, ArrayHasAny


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1970868262

   Take ArrayPopFront, ArrayPopBack, ArrayDims, ArrayDistinct, ArrayElement


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2002881891

   @jayzhan211, Just follow-up to our previous discussion (single func per file), we almost complete `functions-array` migration. Its `kernels` and `udf` source files include more functions by our new pattern. Does it make sense to break them as well by aligning the new pattern and consistency?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2002960214

   @erenavsarogullari
   For consistency, I think we can also change the `invoke` with `make_scalar_function` (probably a better name). It returns ColumnValue::Scalar if length of list is 1. It has no difference currently, since we have no optimization based on Scalar.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1976396644

   > Take ArrayHas, ArrayHasAll, ArrayHasAny
   
   I think this needs to wait until `make_array` is done and also move to rewrite to the optimizer.
   
   Take ArrayIntersect, ArrayUnion, ArrayExcept


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Tangruilin (via GitHub)" <gi...@apache.org>.
Tangruilin commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987263529

   take string module starts_with, to_hex, trim, upper


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "ongchi (via GitHub)" <gi...@apache.org>.
ongchi commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987195735

   Take Tan, Tanh


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "yyy1000 (via GitHub)" <gi...@apache.org>.
yyy1000 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987061171

   Take ArrowTypeOf


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2028033736

   take Atan, Acosh, Asinh, Atanh,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1993593144

   > WDYT on `datafusion/functions-array/src/kernels.rs` (Its size also increases)?
   let's move them out too!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1966386857

   Oh, sorry. I will work on other


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1977562604

   > ListRange likewise, which is kernel function in `function-arrays` crate, instead of converting ArrayRef to Expr, I'm thinking of making array function kernel public. @alamb Do you think we can make those array functions public? And, is importing `function-arrays` to crate `physical-expr` a concern like crate `optimize` is?
   
   I think making physical-exprs depend on datafusion-functions would be unfortunate (as it would still mean there could be dependencies / special cases from built in functions to "built in" user defined functions)
   
   > If neither is a good idea, we may need to rewrite ListIndex and ListRange to array function early.
   
   I agree if this is possible it would be the best approach from my perspective
   
   I also though about about potentially adding the common code to the new `datafusion_common_runtime` crate (that @mustafasrepo  added recently) 
   
   https://github.com/apache/arrow-datafusion/tree/main/datafusion/common_runtime
   
   But I think that would result in a tokio dependency on datafusion-functions which would be unwise. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1977608388

   > I think making physical-exprs depend on datafusion-functions would be unfortunate (as it would still mean there could be dependencies / special cases from built in functions to "built in" user defined functions)
   > 
   > > If neither is a good idea, we may need to rewrite ListIndex and ListRange to array function early.
   > 
   > I agree if this is possible it would be the best approach from my perspective
   
   Hmm. I had the same issue with datebin being referenced in physical-expr - though only for a test which likely could be moved elsewhere or somehow adjusted in some other way. The [draft pr](https://github.com/apache/arrow-datafusion/pull/9435) in fact does just replace it with the moved datebin and thus does import the functions crate. I'll look at updating those tests in some way to remove that dependency.
   
   I also encountered function [references ](https://github.com/apache/arrow-datafusion/blob/a84e5f89bd52d59c78f11fffbab89ed1d418538f/datafusion/sql/src/expr/mod.rs#L173) in the sql crate that I really can't see a way to work around unless we somehow allow SQLExpr to be a part of a ScalarUdf or similar. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987704189

   `ArraySort` and `ArrayDistinct` are also being addressed by following PRs: https://github.com/apache/arrow-datafusion/pull/9551 and https://github.com/apache/arrow-datafusion/pull/9549 so they can also be reflected to above table. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987055095

   Take Flatten


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-2006797382

   Take Ln, Log, Log10, Log2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Epic] Port BuiltInFunctons to `datafusion-functions-*` crates [arrow-datafusion]

Posted by "yyy1000 (via GitHub)" <gi...@apache.org>.
yyy1000 commented on issue #9285:
URL: https://github.com/apache/arrow-datafusion/issues/9285#issuecomment-1987308092

   take Struct in Core


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org