You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Bjørn Jørgensen (Jira)" <ji...@apache.org> on 2021/10/07 20:51:00 UTC

[jira] [Created] (SPARK-36950) Normalize semi-structured data into a flat table.

Bjørn Jørgensen created SPARK-36950:
---------------------------------------

             Summary: Normalize semi-structured data into a flat table.
                 Key: SPARK-36950
                 URL: https://issues.apache.org/jira/browse/SPARK-36950
             Project: Spark
          Issue Type: Wish
          Components: PySpark
    Affects Versions: 3.3.0
            Reporter: Bjørn Jørgensen


Hi, in pandas there is this json_normalize that flat out nested data.

https://github.com/pandas-dev/pandas/blob/v1.3.3/pandas/io/json/_normalize.py#L112-L353 

I have opened a request for this function at koalas. Now there are more people that will have some function over to pyspark.

https://github.com/databricks/koalas/issues/2162

This is also a function that geopandas are using. In the meantime I have found a gist that has code that flattens out the whole dataframe.

https://gist.github.com/nmukerje/e65cde41be85470e4b8dfd9a2d6aed50 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org