You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tanjin Panna (JIRA)" <ji...@apache.org> on 2019/03/04 15:33:00 UTC
[jira] [Comment Edited] (SPARK-27037) Pyspark Row .asDict() cannot
handle MapType with a Struct as the key or value
[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783460#comment-16783460 ]
Tanjin Panna edited comment on SPARK-27037 at 3/4/19 3:32 PM:
--------------------------------------------------------------
[~hyukjin.kwon] what about the key to the map? You can see the {{Row}} is still present.
When [looking at the source code|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L1487], we can see the recursion is only applied to the value, not the key.
was (Author: tanjinp):
[~hyukjin.kwon] what about the key to the map? You can see the {{Row}} is still present.
> Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value
> -----------------------------------------------------------------------------
>
> Key: SPARK-27037
> URL: https://issues.apache.org/jira/browse/SPARK-27037
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.0
> Reporter: Tanjin Panna
> Priority: Major
>
> When we have a tuple as the key or value in a {{MapType}} and call the .{{asDict()}}, we still have a {{Row}} in the output for the key and value:
> {code:java}
> >>> from pyspark.sql import Row
> >>>> df = spark.createDataFrame(
> [
> Row(tuple_map={('hello', True): (1234, 111)}),
> Row(tuple_map={('there', False): (5678, 343)})
> ]
> )
> >>> df.schema
> StructType(List(StructField(tuple_map,MapType(StructType(List(StructField(_1,StringType,true),StructField(_2,BooleanType,true))),StructType(List(StructField(_1,LongType,true),StructField(_2,LongType,true))),true),true)))
> >>> df.show(truncate=False)
> +-------------------------------+
> |tuple_map |
> +-------------------------------+
> |[[hello, true] -> [1234, 111]] |
> |[[there, false] -> [5678, 343]]|
> +-------------------------------+
> >>> df.head().asDict()
> {'tuple_map': {Row(_1=u'hello', _2=True): Row(_1=1234, _2=111)}}
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org