You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Chetan Khatri <ch...@gmail.com> on 2017/03/07 20:58:38 UTC
Issues: Generate JSON with null values in Spark 2.0.x
Hello Dev / Users,
I am working with PySpark Code migration to scala, with Python - Iterating
Spark with dictionary and generating JSON with null is possible with
json.dumps() which will be converted to SparkSQL[Row] but in scala how can
we generate json will null values as a Dataframe ?
Thanks.
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Dongjin Lee <do...@apache.org>.
Hi Chetan,
Sadly, you can not; Spark is configured to ignore the null values when
writing JSON. (check JacksonMessageWriter and find
JsonInclude.Include.NON_NULL from the code.) If you want that
functionality, it would be much better to file the problem to JIRA.
Best,
Dongjin
On Mon, Mar 20, 2017 at 4:44 PM, Chetan Khatri <ch...@gmail.com>
wrote:
> Exactly.
>
> On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <do...@apache.org> wrote:
>
>> Hello Chetan,
>>
>> Could you post some code? If I understood correctly, you are trying to
>> save JSON like:
>>
>> {
>> "first_name": "Dongjin",
>> "last_name: null
>> }
>>
>> not in omitted form, like:
>>
>> {
>> "first_name": "Dongjin"
>> }
>>
>> right?
>>
>> - Dongjin
>>
>> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <
>> chetan.opensource@gmail.com> wrote:
>>
>>> Hello Dev / Users,
>>>
>>> I am working with PySpark Code migration to scala, with Python -
>>> Iterating Spark with dictionary and generating JSON with null is possible
>>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>>> can we generate json will null values as a Dataframe ?
>>>
>>> Thanks.
>>>
>>
>>
>>
>> --
>> *Dongjin Lee*
>>
>>
>> *Software developer in Line+.So interested in massive-scale machine
>> learning.facebook: www.facebook.com/dongjin.lee.kr
>> <http://www.facebook.com/dongjin.lee.kr>linkedin: kr.linkedin.com/in/dongjinleekr
>> <http://kr.linkedin.com/in/dongjinleekr>github:
>> <http://goog_969573159/>github.com/dongjinleekr
>> <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
>> <http://www.twitter.com/dongjinleekr>*
>>
>
>
--
*Dongjin Lee*
*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Dongjin Lee <do...@apache.org>.
Hi Chetan,
Sadly, you can not; Spark is configured to ignore the null values when
writing JSON. (check JacksonMessageWriter and find
JsonInclude.Include.NON_NULL from the code.) If you want that
functionality, it would be much better to file the problem to JIRA.
Best,
Dongjin
On Mon, Mar 20, 2017 at 4:44 PM, Chetan Khatri <ch...@gmail.com>
wrote:
> Exactly.
>
> On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <do...@apache.org> wrote:
>
>> Hello Chetan,
>>
>> Could you post some code? If I understood correctly, you are trying to
>> save JSON like:
>>
>> {
>> "first_name": "Dongjin",
>> "last_name: null
>> }
>>
>> not in omitted form, like:
>>
>> {
>> "first_name": "Dongjin"
>> }
>>
>> right?
>>
>> - Dongjin
>>
>> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <
>> chetan.opensource@gmail.com> wrote:
>>
>>> Hello Dev / Users,
>>>
>>> I am working with PySpark Code migration to scala, with Python -
>>> Iterating Spark with dictionary and generating JSON with null is possible
>>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>>> can we generate json will null values as a Dataframe ?
>>>
>>> Thanks.
>>>
>>
>>
>>
>> --
>> *Dongjin Lee*
>>
>>
>> *Software developer in Line+.So interested in massive-scale machine
>> learning.facebook: www.facebook.com/dongjin.lee.kr
>> <http://www.facebook.com/dongjin.lee.kr>linkedin: kr.linkedin.com/in/dongjinleekr
>> <http://kr.linkedin.com/in/dongjinleekr>github:
>> <http://goog_969573159/>github.com/dongjinleekr
>> <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
>> <http://www.twitter.com/dongjinleekr>*
>>
>
>
--
*Dongjin Lee*
*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Chetan Khatri <ch...@gmail.com>.
Exactly.
On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <do...@apache.org> wrote:
> Hello Chetan,
>
> Could you post some code? If I understood correctly, you are trying to
> save JSON like:
>
> {
> "first_name": "Dongjin",
> "last_name: null
> }
>
> not in omitted form, like:
>
> {
> "first_name": "Dongjin"
> }
>
> right?
>
> - Dongjin
>
> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <chetan.opensource@gmail.com
> > wrote:
>
>> Hello Dev / Users,
>>
>> I am working with PySpark Code migration to scala, with Python -
>> Iterating Spark with dictionary and generating JSON with null is possible
>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>> can we generate json will null values as a Dataframe ?
>>
>> Thanks.
>>
>
>
>
> --
> *Dongjin Lee*
>
>
> *Software developer in Line+.So interested in massive-scale machine
> learning.facebook: www.facebook.com/dongjin.lee.kr
> <http://www.facebook.com/dongjin.lee.kr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>github:
> <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> <http://www.twitter.com/dongjinleekr>*
>
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Chetan Khatri <ch...@gmail.com>.
Exactly.
On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <do...@apache.org> wrote:
> Hello Chetan,
>
> Could you post some code? If I understood correctly, you are trying to
> save JSON like:
>
> {
> "first_name": "Dongjin",
> "last_name: null
> }
>
> not in omitted form, like:
>
> {
> "first_name": "Dongjin"
> }
>
> right?
>
> - Dongjin
>
> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <chetan.opensource@gmail.com
> > wrote:
>
>> Hello Dev / Users,
>>
>> I am working with PySpark Code migration to scala, with Python -
>> Iterating Spark with dictionary and generating JSON with null is possible
>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>> can we generate json will null values as a Dataframe ?
>>
>> Thanks.
>>
>
>
>
> --
> *Dongjin Lee*
>
>
> *Software developer in Line+.So interested in massive-scale machine
> learning.facebook: www.facebook.com/dongjin.lee.kr
> <http://www.facebook.com/dongjin.lee.kr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>github:
> <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> <http://www.twitter.com/dongjinleekr>*
>
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Dongjin Lee <do...@apache.org>.
Hello Chetan,
Could you post some code? If I understood correctly, you are trying to save
JSON like:
{
"first_name": "Dongjin",
"last_name: null
}
not in omitted form, like:
{
"first_name": "Dongjin"
}
right?
- Dongjin
On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <ch...@gmail.com>
wrote:
> Hello Dev / Users,
>
> I am working with PySpark Code migration to scala, with Python - Iterating
> Spark with dictionary and generating JSON with null is possible with
> json.dumps() which will be converted to SparkSQL[Row] but in scala how can
> we generate json will null values as a Dataframe ?
>
> Thanks.
>
--
*Dongjin Lee*
*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*
Re: Issues: Generate JSON with null values in Spark 2.0.x
Posted by Dongjin Lee <do...@apache.org>.
Hello Chetan,
Could you post some code? If I understood correctly, you are trying to save
JSON like:
{
"first_name": "Dongjin",
"last_name: null
}
not in omitted form, like:
{
"first_name": "Dongjin"
}
right?
- Dongjin
On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <ch...@gmail.com>
wrote:
> Hello Dev / Users,
>
> I am working with PySpark Code migration to scala, with Python - Iterating
> Spark with dictionary and generating JSON with null is possible with
> json.dumps() which will be converted to SparkSQL[Row] but in scala how can
> we generate json will null values as a Dataframe ?
>
> Thanks.
>
--
*Dongjin Lee*
*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*