You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2020/02/14 04:17:00 UTC
[jira] [Commented] (SPARK-30781) Missing SortedMap type in pyspark
[ https://issues.apache.org/jira/browse/SPARK-30781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036683#comment-17036683 ]
Hyukjin Kwon commented on SPARK-30781:
--------------------------------------
Currently we don't have the plan to add this given that we now have many higher order functions to workaround this problem,
and the dev overhead by adding a new type - we should consider how it works in Scala, Java, R, Python and SQL.
> Missing SortedMap type in pyspark
> ---------------------------------
>
> Key: SPARK-30781
> URL: https://issues.apache.org/jira/browse/SPARK-30781
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.4.5
> Reporter: Mateusz User
> Priority: Major
> Labels: features
>
> Currently there is only MapType in pyspark API, which does not keep order of key-value map.
>
> *SortedMapType* would fill this gap: a map with sorted key-value pairs (like TreeMap in Java).
>
> For example:
> *SortedMapType* would be very useful when user wants to persist row from DataFrame into Mongo-db.
> row which consist a column with values of an ordered map type:
> col : [1 -> 22, 2 -> 16, 3 -> 25]
> will be persisted as following json:
> { "1": 22, "2": 16, "3": 25 }
>
> instead of MapType which currently results in:
> { "2": 16, "1": 22, "3": 25 }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org