You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by christopher palm <cp...@gmail.com> on 2016/05/24 03:32:38 UTC
Can't Write schema with Optional Group
Hi All,
I see that Parquet-363 made some changes to allow empty schema groups,
however I am testing with this patch and still get the error below on an
optional field.
Cannot write a schema with an empty group: optional group
Is this fix supposed to allow a parquet file to be created when the Avro
schema has optional fields?
Thanks,
Chris
Re: Can't Write schema with Optional Group
Posted by christopher palm <cp...@gmail.com>.
Great catch! This was a deprecated message in the source protobuf, with the
fields removed.
I added in a field into that message and now it works.
Thanks,
Chris
On Tue, May 24, 2016 at 11:02 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:
> Looks like it's failing because records must have at least one field.
>
> On Tue, May 24, 2016 at 9:00 AM, christopher palm <cp...@gmail.com>
> wrote:
>
> > Hi Ryan,
> >
> > Thanks for Clarifying, here is the section of the avro schema that I'm
> > trying to write.
> > {
> > "name": "tireSlippage",
> > "type": ["null", {
> > "type": "record",
> > "name": "TireSlippageEvent",
> > "fields": []
> > }],
> > "default": null
> > },
> >
> > This Schema was created using the ProtobufDatumReader from the original
> > Protobuf class.
> >
> > Is this failing because the type is a record?
> >
> > Thanks,
> > Chris
> >
> > On Tue, May 24, 2016 at 10:02 AM, Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> >
> > > Chris,
> > >
> > > That change allows empty groups when reading, not when writing. For
> > > reading, it allows you to do operations like count without projecting
> any
> > > data columns but there's no corresponding use for writing.
> > >
> > > Optional fields are a different feature, which should work just fine.
> How
> > > are you getting an empty group from an optional field? Avro's pattern
> for
> > > optional is a union with null, so you should have something like this:
> > >
> > > ```json
> > > {
> > > "name" : "some_field",
> > > "type" : [ "null", "string" ],
> > > "default" : null
> > > }
> > > ```
> > >
> > > What is the schema you're trying to write?
> > >
> > > rb
> > >
> > > On Mon, May 23, 2016 at 8:32 PM, christopher palm <cp...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > > I see that Parquet-363 made some changes to allow empty schema
> > groups,
> > > > however I am testing with this patch and still get the error below on
> > an
> > > > optional field.
> > > > Cannot write a schema with an empty group: optional group
> > > >
> > > > Is this fix supposed to allow a parquet file to be created when the
> > Avro
> > > > schema has optional fields?
> > > >
> > > > Thanks,
> > > >
> > > > Chris
> > > >
> > >
> > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> > >
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: Can't Write schema with Optional Group
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Looks like it's failing because records must have at least one field.
On Tue, May 24, 2016 at 9:00 AM, christopher palm <cp...@gmail.com> wrote:
> Hi Ryan,
>
> Thanks for Clarifying, here is the section of the avro schema that I'm
> trying to write.
> {
> "name": "tireSlippage",
> "type": ["null", {
> "type": "record",
> "name": "TireSlippageEvent",
> "fields": []
> }],
> "default": null
> },
>
> This Schema was created using the ProtobufDatumReader from the original
> Protobuf class.
>
> Is this failing because the type is a record?
>
> Thanks,
> Chris
>
> On Tue, May 24, 2016 at 10:02 AM, Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
> > Chris,
> >
> > That change allows empty groups when reading, not when writing. For
> > reading, it allows you to do operations like count without projecting any
> > data columns but there's no corresponding use for writing.
> >
> > Optional fields are a different feature, which should work just fine. How
> > are you getting an empty group from an optional field? Avro's pattern for
> > optional is a union with null, so you should have something like this:
> >
> > ```json
> > {
> > "name" : "some_field",
> > "type" : [ "null", "string" ],
> > "default" : null
> > }
> > ```
> >
> > What is the schema you're trying to write?
> >
> > rb
> >
> > On Mon, May 23, 2016 at 8:32 PM, christopher palm <cp...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > > I see that Parquet-363 made some changes to allow empty schema
> groups,
> > > however I am testing with this patch and still get the error below on
> an
> > > optional field.
> > > Cannot write a schema with an empty group: optional group
> > >
> > > Is this fix supposed to allow a parquet file to be created when the
> Avro
> > > schema has optional fields?
> > >
> > > Thanks,
> > >
> > > Chris
> > >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>
--
Ryan Blue
Software Engineer
Netflix
Re: Can't Write schema with Optional Group
Posted by christopher palm <cp...@gmail.com>.
Hi Ryan,
Thanks for Clarifying, here is the section of the avro schema that I'm
trying to write.
{
"name": "tireSlippage",
"type": ["null", {
"type": "record",
"name": "TireSlippageEvent",
"fields": []
}],
"default": null
},
This Schema was created using the ProtobufDatumReader from the original
Protobuf class.
Is this failing because the type is a record?
Thanks,
Chris
On Tue, May 24, 2016 at 10:02 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:
> Chris,
>
> That change allows empty groups when reading, not when writing. For
> reading, it allows you to do operations like count without projecting any
> data columns but there's no corresponding use for writing.
>
> Optional fields are a different feature, which should work just fine. How
> are you getting an empty group from an optional field? Avro's pattern for
> optional is a union with null, so you should have something like this:
>
> ```json
> {
> "name" : "some_field",
> "type" : [ "null", "string" ],
> "default" : null
> }
> ```
>
> What is the schema you're trying to write?
>
> rb
>
> On Mon, May 23, 2016 at 8:32 PM, christopher palm <cp...@gmail.com>
> wrote:
>
> > Hi All,
> > I see that Parquet-363 made some changes to allow empty schema groups,
> > however I am testing with this patch and still get the error below on an
> > optional field.
> > Cannot write a schema with an empty group: optional group
> >
> > Is this fix supposed to allow a parquet file to be created when the Avro
> > schema has optional fields?
> >
> > Thanks,
> >
> > Chris
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: Can't Write schema with Optional Group
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Chris,
That change allows empty groups when reading, not when writing. For
reading, it allows you to do operations like count without projecting any
data columns but there's no corresponding use for writing.
Optional fields are a different feature, which should work just fine. How
are you getting an empty group from an optional field? Avro's pattern for
optional is a union with null, so you should have something like this:
```json
{
"name" : "some_field",
"type" : [ "null", "string" ],
"default" : null
}
```
What is the schema you're trying to write?
rb
On Mon, May 23, 2016 at 8:32 PM, christopher palm <cp...@gmail.com> wrote:
> Hi All,
> I see that Parquet-363 made some changes to allow empty schema groups,
> however I am testing with this patch and still get the error below on an
> optional field.
> Cannot write a schema with an empty group: optional group
>
> Is this fix supposed to allow a parquet file to be created when the Avro
> schema has optional fields?
>
> Thanks,
>
> Chris
>
--
Ryan Blue
Software Engineer
Netflix