You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Roher (Jira)" <ji...@apache.org> on 2020/02/24 16:39:00 UTC
[jira] [Created] (SPARK-30941) PySpark Row can be instantiated with
duplicate field names
David Roher created SPARK-30941:
-----------------------------------
Summary: PySpark Row can be instantiated with duplicate field names
Key: SPARK-30941
URL: https://issues.apache.org/jira/browse/SPARK-30941
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.4.4
Environment: Ubuntu 18.04
Python 3.6.8
Spark 2.4.4 (installed via binary from website)
Reporter: David Roher
It is possible to create a Row that has fields with the same name when calling `collect()` after a join. Given that the Row constructor itself doesn't allow this, this seems to be undesired behavior.
This can possibly cause correctness issues because different ways of getting values produce different results: {{__getitem__}} will return the leftmost value, while {{asDict()}} will return the rightmost value (because the former uses an index search and the latter uses a dictionary generator).
{{>>> manual_output_row = Row(a=1, b=1, b=2)}}
{{ File "<stdin>", line 1}}
{{SyntaxError: keyword argument repeated}}
{{>>> input_rows = Row(a=1, b=1), Row(a=1, b=2)}}
{{>>> df1, df2 = (spark.createDataFrame([r]) for r in input_rows)}}
{{>>> df3 = df1.join(df2, "a")}}
{{>>> output_row = df3.collect()[0]}}
{{>>> output_row}}
{{Row(a=1, b=1, b=2)}}
{{>>> output_row["b"]}}
{{1}}
{{>>> output_row.asDict()["b"]}}
{{2}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org