You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ken Tore Tallakstad <ta...@gmail.com> on 2018/10/25 06:58:23 UTC
PutParquet - Array contains null element at 0
Hi,
We have an issue with PutParquet (NiFi 1.7.1), well with the parquet lib to
be precise, and array type data containing null values.
This is a schema snippet of the field in question:
{
"name": "adresse",
"type" : ["null", { "type" : "array", "items" :
["null","string"], "default": null } ], "default": null
},
And a corresponding data example:
"adresse" : [ null, "value1" ],
"adresse" : [ null, "value2" ],
"adresse" : [ "value3", null, "value4" ],
Avro does not seem to have a problem with this and all our records pass,
but Put parquet fails with the following error: "Array contains a null
element at X".
Apparently there is a parquet config to allow
this: parquet.avro.write-old-list-structure=false. Any tips on how to set
it? Are there any other ways around this, besides stripping the raw data of
nulls in arrays?
Thanks!
KT :)
[image: parq1.png]
[image: parq2.png]
Re: PutParquet - Array contains null element at 0
Posted by Bryan Bende <bb...@gmail.com>.
Currently I don't think there is a way that config value can be set without
a code change, but if you want to create a JIRA it would probably make
sense to expose that as a property in the processor to toggle between true
and false, or we can also allow make it so that any dynamic properties get
passed through to the Parquet writer's conf.
On Thu, Oct 25, 2018 at 2:58 AM Ken Tore Tallakstad <ta...@gmail.com>
wrote:
> Hi,
>
> We have an issue with PutParquet (NiFi 1.7.1), well with the parquet lib
> to be precise, and array type data containing null values.
> This is a schema snippet of the field in question:
> {
> "name": "adresse",
> "type" : ["null", { "type" : "array", "items" :
> ["null","string"], "default": null } ], "default": null
> },
>
> And a corresponding data example:
> "adresse" : [ null, "value1" ],
> "adresse" : [ null, "value2" ],
> "adresse" : [ "value3", null, "value4" ],
>
> Avro does not seem to have a problem with this and all our records pass,
> but Put parquet fails with the following error: "Array contains a null
> element at X".
>
> Apparently there is a parquet config to allow
> this: parquet.avro.write-old-list-structure=false. Any tips on how to set
> it? Are there any other ways around this, besides stripping the raw data of
> nulls in arrays?
>
> Thanks!
>
> KT :)
>
>
> [image: parq1.png]
>
>
> [image: parq2.png]
>