You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/10/08 00:27:00 UTC
[jira] [Commented] (SPARK-36950) Normalize semi-structured data
into a flat table.
[ https://issues.apache.org/jira/browse/SPARK-36950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425873#comment-17425873 ]
Hyukjin Kwon commented on SPARK-36950:
--------------------------------------
Thanks [~bjornjorgensen]
> Normalize semi-structured data into a flat table.
> -------------------------------------------------
>
> Key: SPARK-36950
> URL: https://issues.apache.org/jira/browse/SPARK-36950
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.3.0
> Reporter: Bjørn Jørgensen
> Priority: Major
>
> Hi, in pandas there is this json_normalize that flat out nested data.
> https://github.com/pandas-dev/pandas/blob/v1.3.3/pandas/io/json/_normalize.py#L112-L353
> I have opened a request for this function at koalas. Now there are more people that will have some function over to pyspark.
> https://github.com/databricks/koalas/issues/2162
> This is also a function that geopandas are using. In the meantime I have found a gist that has code that flattens out the whole dataframe.
> https://gist.github.com/nmukerje/e65cde41be85470e4b8dfd9a2d6aed50
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org