You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Karthick Nk <kc...@gmail.com> on 2022/10/04 03:29:42 UTC

Converting None/Null into json in pyspark

Hi all,

I need to convert pyspark dataframe into json .

While converting , if all rows values are null/None for that particular
column that column is getting removed from data.

Could you suggest a way to do this. I need to convert dataframe into json
with columns.

Thanks

Re: Converting None/Null into json in pyspark

Posted by Yeachan Park <ye...@gmail.com>.

You can try this (replace spark with whatever variable your sparksession
is): spark.conf.set("spark.sql.jsonGenerator.ignoreNullFields", False)

On Tue, Oct 4, 2022 at 4:55 PM Karthick Nk <kc...@gmail.com> wrote:

> Thanks
> I am using Pyspark in databricks, I have seen through multiple reference
> but I couldn't find the exact snippet. Could you share a sample snippet for
> the same how do I set that property.
>
> My step:
> df = df.selectExpr(f'to_json(struct(*)) as json_data')
>
> On Tue, Oct 4, 2022 at 10:57 AM Yeachan Park <ye...@gmail.com> wrote:
>
>> Hi,
>>
>> There's a config option for this. Try setting this to false in your spark
>> conf.
>>
>> spark.sql.jsonGenerator.ignoreNullFields
>>
>> On Tuesday, October 4, 2022, Karthick Nk <kc...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I need to convert pyspark dataframe into json .
>>>
>>> While converting , if all rows values are null/None for that particular
>>> column that column is getting removed from data.
>>>
>>> Could you suggest a way to do this. I need to convert dataframe into
>>> json with columns.
>>>
>>> Thanks
>>>
>>

Re: Converting None/Null into json in pyspark

Posted by Karthick Nk <kc...@gmail.com>.

 Thanks
I am using Pyspark in databricks, I have seen through multiple reference
but I couldn't find the exact snippet. Could you share a sample snippet for
the same how do I set that property.

My step:
df = df.selectExpr(f'to_json(struct(*)) as json_data')

On Tue, Oct 4, 2022 at 10:57 AM Yeachan Park <ye...@gmail.com> wrote:

> Hi,
>
> There's a config option for this. Try setting this to false in your spark
> conf.
>
> spark.sql.jsonGenerator.ignoreNullFields
>
> On Tuesday, October 4, 2022, Karthick Nk <kc...@gmail.com> wrote:
>
>> Hi all,
>>
>> I need to convert pyspark dataframe into json .
>>
>> While converting , if all rows values are null/None for that particular
>> column that column is getting removed from data.
>>
>> Could you suggest a way to do this. I need to convert dataframe into json
>> with columns.
>>
>> Thanks
>>
>

Re: Converting None/Null into json in pyspark

Posted by Yeachan Park <ye...@gmail.com>.

Hi,

There's a config option for this. Try setting this to false in your spark
conf.

spark.sql.jsonGenerator.ignoreNullFields

On Tuesday, October 4, 2022, Karthick Nk <kc...@gmail.com> wrote:

> Hi all,
>
> I need to convert pyspark dataframe into json .
>
> While converting , if all rows values are null/None for that particular
> column that column is getting removed from data.
>
> Could you suggest a way to do this. I need to convert dataframe into json
> with columns.
>
> Thanks
>