You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daniel deCordoba (Jira)" <ji...@apache.org> on 2022/04/08 20:59:00 UTC

[jira] [Created] (SPARK-38839) Creating a struct with a float inside

Daniel deCordoba created SPARK-38839:
----------------------------------------

             Summary: Creating a struct with a float inside 
                 Key: SPARK-38839
                 URL: https://issues.apache.org/jira/browse/SPARK-38839
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.2.1
            Reporter: Daniel deCordoba


When creating a dataframe using createDataFrame that contains a float inside a struct, the float is set to null. This only happens if using a list of dictionaries as data type, if I use a list of Rows it works fine:

```python
data = [\{"MyStruct": {"MyInt": 10, "MyFloat": 10.1}, "MyFloat": 10.1}]

spark.createDataFrame(data).show()
# +-------+------------------------------+
# |MyFloat|MyStruct                      |
# +-------+------------------------------+
# |10.1   |\{MyInt -> 10, MyFloat -> null}|
# +-------+------------------------------+

 

data = [Row(MyStruct=Row(MyInt=10, MyFloat=10.1), MyFloat=10.1)]

spark.createDataFrame(data).show()
# +-------+------------------------------+
# |MyFloat|MyStruct                      |
# +-------+------------------------------+
# |10.1   |\{MyInt -> 10, MyFloat -> 10.1}|
# +-------+------------------------------+

```
Note MyFloat inside MyStruct is set to null in the first example. Interestingly enough, when I do the same with Row, or if I specify the schema, then this does not happen (second example).



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org