You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/09/10 21:35:00 UTC
[jira] [Assigned] (SPARK-32846) Support createDataFrame from an RDD
of pd.DataFrames
[ https://issues.apache.org/jira/browse/SPARK-32846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-32846:
------------------------------------
Assignee: (was: Apache Spark)
> Support createDataFrame from an RDD of pd.DataFrames
> ----------------------------------------------------
>
> Key: SPARK-32846
> URL: https://issues.apache.org/jira/browse/SPARK-32846
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, SQL
> Affects Versions: 3.0.1
> Reporter: Linar Savion
> Priority: Minor
> Labels: arrow, pandas, sql
>
> Add support to createDataFrame from a distributed collection of pandas.DataFrames by converting the RDD of pd.DFs to an RDD of arrow records batches, then directly creating the spark DataFrame from it.
>
> Performance is significantly better (vectorized) than creating a spark DF by converting each df to a list of rows, similar to the improvement of SPARK-20791.
>
> Initial example & benchmark for older spark versions: [https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5|https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5,]
>
> I'm currently working on a PR and will post it soon.
>
> Extends the work done in:
> https://issues.apache.org/jira/browse/SPARK-20791
> https://issues.apache.org/jira/browse/SPARK-23030
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org