You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/05/31 20:14:00 UTC

[jira] [Updated] (SPARK-48493) Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance

     [ https://issues.apache.org/jira/browse/SPARK-48493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-48493:
-----------------------------------
    Labels: pull-request-available  (was: )

> Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-48493
>                 URL: https://issues.apache.org/jira/browse/SPARK-48493
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.0.0
>            Reporter: Luca Canali
>            Priority: Minor
>              Labels: pull-request-available
>
>  
> This proposes an enhancement to the Python Datasource Reader by adding an option to yield Arrow batches directly, significantly boosting performance compared to using tuples or Rows. This implementation uses the existing work with MapInArrow (see SPARK-46253 ).
> In tests with a custom Python Datasource for High Energy Physics data (ROOT format reader), using Arrow batches has demonstrated an 8x speed increase over the traditional method of feeding data via tuples.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org