[GitHub] [spark] bjornjorgensen edited a comment on pull request #35296: [SPARK-38067][PYTHON] Preserve columns with all None values when saved to JSON file.

[GitBox <gi...@apache.org>]

bjornjorgensen edited a comment on pull request #35296:
URL: https://github.com/apache/spark/pull/35296#issuecomment-1025144269


   @zero323 This PR won't have any impact on your first example because there is at least one value that is not None.  In your second example it will be without this PR. 
   
   ```bash
   df = ps.DataFrame.from_dict({"id": [1, 2], "val": [np.nan, np.nan]})
   df
   id val
   0 1 NaN
   1 2 NaN
   
   df.to_json("testdf.json", num_files=1)
   
   df2 = ps.read_json("testdf.json/*")
   df2
   
   id val
   0 1 None
   1 2 None
   
   df2.to_json("testdf2.json", ignoreNullFields=True, num_files=1)
   
   
   df3 = ps.read_json("testdf2.json/*")
   
   df3
   
   id
   0 1
   1 2
   ```
   
   I will change what we agree on, but for now I think this has something to do with None values.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org