[I] Extend native Arrow UDF path to grouped-aggregate, window, and applyInArrow Python operators [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #5123:
URL: https://github.com/apache/datafusion-comet/issues/5123

   ### What is the problem the feature request solves?
   
   Comet has an opt-in native path (`spark.comet.exec.pyarrowUdf.enabled`) that keeps data in Arrow format across `mapInArrow` / `mapInPandas`, avoiding Spark's Arrow to row to Arrow round trip. #4234 extends this to scalar pandas UDFs (`ArrowEvalPythonExec`) and `applyInPandas` (`FlatMapGroupsInPandasExec`).
   
   The remaining Arrow-based Python operators still fall back, even though they exchange Arrow batches with the Python worker and the same optimization applies in principle:
   
   - `AggregateInPandasExec` / `ArrowAggregatePythonExec` (grouped-aggregate pandas UDFs, renamed in Spark 4.1)
   - `WindowInPandasExec` / `ArrowWindowPythonExec` (window pandas UDFs, renamed in Spark 4.1)
   - `FlatMapGroupsInArrowExec` (`applyInArrow`, Spark 4.0+)
   - `FlatMapCoGroupsInPandasExec` / `FlatMapCoGroupsInArrowExec` (`cogroup(...).applyInPandas` / `applyInArrow`)
   - `ArrowEvalPythonUDTFExec` (Arrow-based Python UDTFs, Spark 3.5+)
   
   For Python-heavy analytics workloads these operators sit in the middle of otherwise native stages, so each one both pays the row round trip and splits the native stage.
   
   ### Describe the potential solution
   
   Extend the pyarrowUdf native path to these operators after #4234 lands, in roughly the order listed, since grouped-aggregate and window pandas UDFs are the most common in analytics code. The grouped variants additionally need the grouping/partitioning handled natively, which the `FlatMapGroupsInPandasExec` work in #4234 establishes the pattern for.
   
   Pickled (non-Arrow) UDF operators are out of scope since there is no columnar boundary to preserve. `AttachDistributedSequenceExec` is also out of scope here.
   
   ### Additional context
   
   Related: #4234, the PyArrow UDF user guide, #4384 (fuzz harness for the pyarrow UDF vector-copy path).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Extend native Arrow UDF path to grouped-aggregate, window, and applyInArrow Python operators [datafusion-comet]

["peterxcli (via GitHub)" <gi...@apache.org>]

peterxcli commented on issue #5123:
URL: https://github.com/apache/datafusion-comet/issues/5123#issuecomment-5129883273

   let me take a stab at this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org