You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ion-elgreco (via GitHub)" <gi...@apache.org> on 2024/09/20 08:51:03 UTC

[I] RFC: Refactor functions into Expressions namespaces and functions on the Expr directly [datafusion-python]

ion-elgreco opened a new issue, #876:
URL: https://github.com/apache/datafusion-python/issues/876

   One thing I disliked a lot about pyspark was the Functions. Since most of if not all DataFusion.functions take in an expression and return an expression, we could rework this in the Expr namespaces, this should help finding the right expressions much easier, some examples:
   
   - `col().dt.current_date()`
   - `col().list.ndims()`
   
   I am happy to start working on this and creating a full list of how the Expr and Expr.namespaces can look like


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] RFC: Refactor functions into Expressions namespaces and functions on the Expr directly [datafusion-python]

Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #876:
URL: https://github.com/apache/datafusion-python/issues/876#issuecomment-2363581681

   > My primary concerns about doing this are:
   > 
   > * API churn rate. If we keep changing our interface too rapidly between versions it will turn people off the project. So if we did this I'd probably support defining them on the expr but also having the functions available - one to call the other, almost as an alias.
   
   I think this can work but with the goal to deprecate the functions after X releases/months. One thing that is problematic for example with PySpark codebases is that there is no consistency due to the aliases. I don't think constant change is an issue, unless you don't document well how to change.
   
   > * Familiarity. Like it or not, many if not most, of the people who will be coming to the project will be coming from a pyspark/pandas/polars background. The layout we currently have will be more familiar _in some places_ which I think will lead to greater adoption.
   
   In some way this might speak more to PySpark users the current API, but arguably and I think many will agree if you come from polars and pandas the current API isn't close to their familiarity. 
   
   > * Diverging too much from the datafusion core approach. I think some level of difference is expected since rust and python do operate very differently. However if we diverge too much, it makes maintenance harder and harder to onboard new contributors. This is a smaller concern.
   > 
   
   > That being said, I have found that some of the functions I expected to be on `Expr` were in `functions` and I do find the interface to be somewhat non-intuitive. My proposal would be to handle these on a case by case basis of what should be where.
   
   I can make a draft of mapping Functions -> Expr, so we can get a full picture on how this will look like


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] RFC: Refactor functions into Expressions namespaces and functions on the Expr directly [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #876:
URL: https://github.com/apache/datafusion-python/issues/876#issuecomment-2363536222

   My primary concerns about doing this are:
   
   - API churn rate. If we keep changing our interface too rapidly between versions it will turn people off the project. So if we did this I'd probably support defining them on the expr but also having the functions available - one to call the other, almost as an alias.
   - Familiarity. Like it or not, many if not most, of the people who will be coming to the project will be coming from a pyspark/pandas/polars background. The layout we currently have will be more familiar *in some places* which I think will lead to greater adoption.
   - Diverging too much from the datafusion core approach. I think some level of difference is expected since rust and python do operate very differently. However if we diverge too much, it makes maintenance harder and harder to onboard new contributors. This is a smaller concern.
   
   That being said, I have found that some of the functions I expected to be on `Expr` were in `functions` and I do find the interface to be somewhat non-intuitive. My proposal would be to handle these on a case by case basis of what should be where.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org