You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Douglas Moore (Jira)" <ji...@apache.org> on 2022/11/01 17:00:00 UTC
[jira] [Comment Edited] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627251#comment-17627251 ]
Douglas Moore edited comment on SPARK-37697 at 11/1/22 4:59 PM:
----------------------------------------------------------------
Sorry, I don't know what's going on...
This works:
{noformat}
spark.createDataFrame(np.array([[1, 2], [3, 4]])).show(){noformat}
This doesn't:
{code:java}
spark.createDataFrame(np.arange(4.).reshape([2,3])).show(){code}
Both have same type() and same .shape
was (Author: douglas.moore@databricks.com):
Sorry, I don't know what's going on...
This works: `spark.createDataFrame(np.array([[1, 2], [3, 4]])).show()`
This doesn't: `spark.createDataFrame(np.arange(4.).reshape([2,3])).show()`
Both have same `type()` and same `.shape`
> Make it easier to convert numpy arrays to Spark Dataframes
> ----------------------------------------------------------
>
> Key: SPARK-37697
> URL: https://issues.apache.org/jira/browse/SPARK-37697
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.1.2
> Reporter: Douglas Moore
> Priority: Major
> Attachments: image-2022-10-31-22-49-37-356.png
>
>
> Make it easier to convert numpy arrays to dataframes.
> Often we receive errors:
>
> {code:java}
> df = spark.createDataFrame(numpy.arange(10))
> Can not infer schema for type: <class 'numpy.int64'>
> {code}
>
> OR
> {code:java}
> df = spark.createDataFrame(numpy.arange(10.))
> Can not infer schema for type: <class 'numpy.float64'>
> {code}
>
> Today (Spark 3.x) we have to:
> {code:java}
> spark.createDataFrame(pd.DataFrame(numpy.arange(10.))) {code}
> Make this easier with a direct conversion from Numpy arrays to Spark Dataframes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org