You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sid <fl...@gmail.com> on 2022/02/23 17:56:42 UTC
Unable to display JSON records with null values
Hello experts,
I have a JSON data like below:
[
{
"123": {
"Party1": {
"FIRSTNAMEBEN": "ABC",
"ALIASBEN": "",
"RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
"DATEOFBIRTH": "7/Oct/1969"
},
"Party2": {
"FIRSTNAMEBEN": "ABCC",
"ALIASBEN": "",
"RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
"DATEOFBIRTH": "7/Oct/1969"
}
},
"GeneratedTime": "2022-01-30 03:09:26"
},
{
"456": {
"Party1": {
"FIRSTNAMEBEN": "ABCD",
"ALIASBEN": "",
"RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
"DATEOFBIRTH": "7/Oct/1969"
},
"Party2": {
"FIRSTNAMEBEN": "ABCDD",
"ALIASBEN": "",
"RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
"DATEOFBIRTH": "7/Oct/1969"
},
"Party3": {
"FIRSTNAMEBEN": "ABCDDE",
"ALIASBEN": "",
"RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
"DATEOFBIRTH": "7/Oct/1969"
}
},
"GeneratedTime": "2022-01-30 03:09:26"
},
{
"345": {
},
"GeneratedTime": "2022-01-30 03:09:26"
}
]
However, when I try to display this JSON using below code, it doesn't show
the blank records. In my case I don't get any records for 345 since it is
null but I want to display it in the final flattened dataset.
val df = spark.read.option("multiline",
true).json("/home/siddhesh/Documents/nested_json.json")
Spark version:3.1.1
Thanks,
Sid
Re: Unable to display JSON records with null values
Posted by Sid <fl...@gmail.com>.
Okay. So what should I do if I get such data?
On Wed, Feb 23, 2022 at 11:59 PM Sean Owen <sr...@gmail.com> wrote:
> There is no record "345" here it seems, right? it's not that it exists and
> has null fields; it's invalid w.r.t. the schema that the rest suggests.
>
> On Wed, Feb 23, 2022 at 11:57 AM Sid <fl...@gmail.com> wrote:
>
>> Hello experts,
>>
>> I have a JSON data like below:
>>
>> [
>> {
>> "123": {
>> "Party1": {
>> "FIRSTNAMEBEN": "ABC",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party2": {
>> "FIRSTNAMEBEN": "ABCC",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> }
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> },
>> {
>> "456": {
>> "Party1": {
>> "FIRSTNAMEBEN": "ABCD",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party2": {
>> "FIRSTNAMEBEN": "ABCDD",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party3": {
>> "FIRSTNAMEBEN": "ABCDDE",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> }
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> },
>> {
>> "345": {
>>
>>
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> }
>> ]
>>
>> However, when I try to display this JSON using below code, it doesn't
>> show the blank records. In my case I don't get any records for 345 since it
>> is null but I want to display it in the final flattened dataset.
>>
>> val df = spark.read.option("multiline",
>> true).json("/home/siddhesh/Documents/nested_json.json")
>>
>> Spark version:3.1.1
>>
>> Thanks,
>> Sid
>>
>
Re: Unable to display JSON records with null values
Posted by Sean Owen <sr...@gmail.com>.
There is no record "345" here it seems, right? it's not that it exists and
has null fields; it's invalid w.r.t. the schema that the rest suggests.
On Wed, Feb 23, 2022 at 11:57 AM Sid <fl...@gmail.com> wrote:
> Hello experts,
>
> I have a JSON data like below:
>
> [
> {
> "123": {
> "Party1": {
> "FIRSTNAMEBEN": "ABC",
> "ALIASBEN": "",
> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
> "DATEOFBIRTH": "7/Oct/1969"
> },
> "Party2": {
> "FIRSTNAMEBEN": "ABCC",
> "ALIASBEN": "",
> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
> "DATEOFBIRTH": "7/Oct/1969"
> }
> },
> "GeneratedTime": "2022-01-30 03:09:26"
> },
> {
> "456": {
> "Party1": {
> "FIRSTNAMEBEN": "ABCD",
> "ALIASBEN": "",
> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
> "DATEOFBIRTH": "7/Oct/1969"
> },
> "Party2": {
> "FIRSTNAMEBEN": "ABCDD",
> "ALIASBEN": "",
> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
> "DATEOFBIRTH": "7/Oct/1969"
> },
> "Party3": {
> "FIRSTNAMEBEN": "ABCDDE",
> "ALIASBEN": "",
> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
> "DATEOFBIRTH": "7/Oct/1969"
> }
> },
> "GeneratedTime": "2022-01-30 03:09:26"
> },
> {
> "345": {
>
>
> },
> "GeneratedTime": "2022-01-30 03:09:26"
> }
> ]
>
> However, when I try to display this JSON using below code, it doesn't show
> the blank records. In my case I don't get any records for 345 since it is
> null but I want to display it in the final flattened dataset.
>
> val df = spark.read.option("multiline",
> true).json("/home/siddhesh/Documents/nested_json.json")
>
> Spark version:3.1.1
>
> Thanks,
> Sid
>