You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Stefán Baxter <st...@activitystream.com> on 2015/10/16 15:00:17 UTC
Avro support
Hi,
I'm changing our logging to use the Avro format and there I'm relying on
Maps and Records (f.ex. union ["null","RecordType"]) in addition to
primitive types.
I'm getting this error in Drill when I try to query the test file:
Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type must be
of the format : ["null", "some-type"]
This error message provides no clues to which field are involved here but
I'm hoping that this does not mean that sub structures, available in both
JSON and Parquet, are not available in Avro as Records.
Can someone please shed some light on this.
Regards,
-Stefan
Re: Avro support
Posted by Jacques Nadeau <ja...@dremio.com>.
Can you post a JIRA around the example avro schema and the poor error
message?
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Sat, Oct 17, 2015 at 2:20 AM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> This turned out to be a incomplete union:
>
>
> - {"name": "market", "type": ["string"]}
>
> Avro tools ignored it and created the classes as if this was a normal
> string field.
>
> I would really appreciate it if the error messages Drill produces would be
> structured in a more helpful way.
>
> Knowing what went wrong is fine, knowing why it went wrong is even better
> and I think that in this case it should be easy to include a piece of the
> header that is being parse/processed.
>
> Regards,
> -Stefan
>
>
>
> On Fri, Oct 16, 2015 at 4:45 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi Kamesh,
> >
> > I pulled, made and deployed 1.2-SNAPSHOT (master/head/"Latest commit
> > 12fb9a
> > <
> https://github.com/apache/drill/commit/12fb9ac11e25a2bbd9fff5d9cdb6a355b96ffadb
> >
> > 3 days ago") producing the same results.
> >
> > I can not see a RC3 Branch, am I missing something?
> >
> > Regards,
> > -Stefan
> >
> >
> >
> >
> > On Fri, Oct 16, 2015 at 4:31 PM, Stefán Baxter <
> stefan@activitystream.com>
> > wrote:
> >
> >> Hi Kamesh,
> >>
> >> I must be lagging behind then, let me double check.
> >>
> >> Regards,
> >> -Stefan
> >>
> >> On Fri, Oct 16, 2015 at 4:23 PM, Kamesh <ka...@gmail.com>
> wrote:
> >>
> >>> Hi Stefan,
> >>> Do you have latest code changes with you (1.2-SNAPSHOT)? Could you
> >>> please
> >>> try RC3 build.
> >>> Yes, it supports nested records.
> >>>
> >>>
> >>> On Fri, Oct 16, 2015 at 8:28 PM, Stefán Baxter <
> >>> stefan@activitystream.com>
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
> >>> >
> >>> > Should it be supporting Records as a data type? (use to map sub
> >>> > structures).
> >>> >
> >>> > Regards,
> >>> > -Stefán
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Hi Stefan,
> >>> > > We have enhanced Drill Avro Reader by supporting complex data
> types
> >>> like
> >>> > > Union, Map etc.. This is targeted for 1.2.0 release. This release
> >>> will
> >>> > > happen in a day or in a couple days.
> >>> > >
> >>> > > On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <
> >>> > stefan@activitystream.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Hi,
> >>> > > >
> >>> > > > I'm changing our logging to use the Avro format and there I'm
> >>> relying
> >>> > on
> >>> > > > Maps and Records (f.ex. union ["null","RecordType"]) in addition
> to
> >>> > > > primitive types.
> >>> > > >
> >>> > > > I'm getting this error in Drill when I try to query the test
> file:
> >>> > > >
> >>> > > > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union
> type
> >>> > must
> >>> > > be
> >>> > > > of the format : ["null", "some-type"]
> >>> > > >
> >>> > > >
> >>> > > > This error message provides no clues to which field are involved
> >>> here
> >>> > but
> >>> > > > I'm hoping that this does not mean that sub structures, available
> >>> in
> >>> > both
> >>> > > > JSON and Parquet, are not available in Avro as Records.
> >>> > > >
> >>> > > > Can someone please shed some light on this.
> >>> > > >
> >>> > > > Regards,
> >>> > > > -Stefan
> >>> > > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Kamesh.
> >>> > >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Kamesh.
> >>>
> >>
> >>
> >
>
Re: Avro support
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
This turned out to be a incomplete union:
- {"name": "market", "type": ["string"]}
Avro tools ignored it and created the classes as if this was a normal
string field.
I would really appreciate it if the error messages Drill produces would be
structured in a more helpful way.
Knowing what went wrong is fine, knowing why it went wrong is even better
and I think that in this case it should be easy to include a piece of the
header that is being parse/processed.
Regards,
-Stefan
On Fri, Oct 16, 2015 at 4:45 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi Kamesh,
>
> I pulled, made and deployed 1.2-SNAPSHOT (master/head/"Latest commit
> 12fb9a
> <https://github.com/apache/drill/commit/12fb9ac11e25a2bbd9fff5d9cdb6a355b96ffadb>
> 3 days ago") producing the same results.
>
> I can not see a RC3 Branch, am I missing something?
>
> Regards,
> -Stefan
>
>
>
>
> On Fri, Oct 16, 2015 at 4:31 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
>> Hi Kamesh,
>>
>> I must be lagging behind then, let me double check.
>>
>> Regards,
>> -Stefan
>>
>> On Fri, Oct 16, 2015 at 4:23 PM, Kamesh <ka...@gmail.com> wrote:
>>
>>> Hi Stefan,
>>> Do you have latest code changes with you (1.2-SNAPSHOT)? Could you
>>> please
>>> try RC3 build.
>>> Yes, it supports nested records.
>>>
>>>
>>> On Fri, Oct 16, 2015 at 8:28 PM, Stefán Baxter <
>>> stefan@activitystream.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
>>> >
>>> > Should it be supporting Records as a data type? (use to map sub
>>> > structures).
>>> >
>>> > Regards,
>>> > -Stefán
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Stefan,
>>> > > We have enhanced Drill Avro Reader by supporting complex data types
>>> like
>>> > > Union, Map etc.. This is targeted for 1.2.0 release. This release
>>> will
>>> > > happen in a day or in a couple days.
>>> > >
>>> > > On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <
>>> > stefan@activitystream.com>
>>> > > wrote:
>>> > >
>>> > > > Hi,
>>> > > >
>>> > > > I'm changing our logging to use the Avro format and there I'm
>>> relying
>>> > on
>>> > > > Maps and Records (f.ex. union ["null","RecordType"]) in addition to
>>> > > > primitive types.
>>> > > >
>>> > > > I'm getting this error in Drill when I try to query the test file:
>>> > > >
>>> > > > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type
>>> > must
>>> > > be
>>> > > > of the format : ["null", "some-type"]
>>> > > >
>>> > > >
>>> > > > This error message provides no clues to which field are involved
>>> here
>>> > but
>>> > > > I'm hoping that this does not mean that sub structures, available
>>> in
>>> > both
>>> > > > JSON and Parquet, are not available in Avro as Records.
>>> > > >
>>> > > > Can someone please shed some light on this.
>>> > > >
>>> > > > Regards,
>>> > > > -Stefan
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Kamesh.
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> Kamesh.
>>>
>>
>>
>
Re: Avro support
Posted by Stefán Baxter <st...@activitystream.com>.
Hi Kamesh,
I pulled, made and deployed 1.2-SNAPSHOT (master/head/"Latest commit 12fb9a
<https://github.com/apache/drill/commit/12fb9ac11e25a2bbd9fff5d9cdb6a355b96ffadb>
3 days ago") producing the same results.
I can not see a RC3 Branch, am I missing something?
Regards,
-Stefan
On Fri, Oct 16, 2015 at 4:31 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi Kamesh,
>
> I must be lagging behind then, let me double check.
>
> Regards,
> -Stefan
>
> On Fri, Oct 16, 2015 at 4:23 PM, Kamesh <ka...@gmail.com> wrote:
>
>> Hi Stefan,
>> Do you have latest code changes with you (1.2-SNAPSHOT)? Could you please
>> try RC3 build.
>> Yes, it supports nested records.
>>
>>
>> On Fri, Oct 16, 2015 at 8:28 PM, Stefán Baxter <stefan@activitystream.com
>> >
>> wrote:
>>
>> > Hi,
>> >
>> > Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
>> >
>> > Should it be supporting Records as a data type? (use to map sub
>> > structures).
>> >
>> > Regards,
>> > -Stefán
>> >
>> >
>> >
>> >
>> > On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com>
>> wrote:
>> >
>> > > Hi Stefan,
>> > > We have enhanced Drill Avro Reader by supporting complex data types
>> like
>> > > Union, Map etc.. This is targeted for 1.2.0 release. This release will
>> > > happen in a day or in a couple days.
>> > >
>> > > On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <
>> > stefan@activitystream.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I'm changing our logging to use the Avro format and there I'm
>> relying
>> > on
>> > > > Maps and Records (f.ex. union ["null","RecordType"]) in addition to
>> > > > primitive types.
>> > > >
>> > > > I'm getting this error in Drill when I try to query the test file:
>> > > >
>> > > > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type
>> > must
>> > > be
>> > > > of the format : ["null", "some-type"]
>> > > >
>> > > >
>> > > > This error message provides no clues to which field are involved
>> here
>> > but
>> > > > I'm hoping that this does not mean that sub structures, available in
>> > both
>> > > > JSON and Parquet, are not available in Avro as Records.
>> > > >
>> > > > Can someone please shed some light on this.
>> > > >
>> > > > Regards,
>> > > > -Stefan
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kamesh.
>> > >
>> >
>>
>>
>>
>> --
>> Kamesh.
>>
>
>
Re: Avro support
Posted by Stefán Baxter <st...@activitystream.com>.
Hi Kamesh,
I must be lagging behind then, let me double check.
Regards,
-Stefan
On Fri, Oct 16, 2015 at 4:23 PM, Kamesh <ka...@gmail.com> wrote:
> Hi Stefan,
> Do you have latest code changes with you (1.2-SNAPSHOT)? Could you please
> try RC3 build.
> Yes, it supports nested records.
>
>
> On Fri, Oct 16, 2015 at 8:28 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
> >
> > Should it be supporting Records as a data type? (use to map sub
> > structures).
> >
> > Regards,
> > -Stefán
> >
> >
> >
> >
> > On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com> wrote:
> >
> > > Hi Stefan,
> > > We have enhanced Drill Avro Reader by supporting complex data types
> like
> > > Union, Map etc.. This is targeted for 1.2.0 release. This release will
> > > happen in a day or in a couple days.
> > >
> > > On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <
> > stefan@activitystream.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm changing our logging to use the Avro format and there I'm relying
> > on
> > > > Maps and Records (f.ex. union ["null","RecordType"]) in addition to
> > > > primitive types.
> > > >
> > > > I'm getting this error in Drill when I try to query the test file:
> > > >
> > > > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type
> > must
> > > be
> > > > of the format : ["null", "some-type"]
> > > >
> > > >
> > > > This error message provides no clues to which field are involved here
> > but
> > > > I'm hoping that this does not mean that sub structures, available in
> > both
> > > > JSON and Parquet, are not available in Avro as Records.
> > > >
> > > > Can someone please shed some light on this.
> > > >
> > > > Regards,
> > > > -Stefan
> > > >
> > >
> > >
> > >
> > > --
> > > Kamesh.
> > >
> >
>
>
>
> --
> Kamesh.
>
Re: Avro support
Posted by Kamesh <ka...@gmail.com>.
Hi Stefan,
Do you have latest code changes with you (1.2-SNAPSHOT)? Could you please
try RC3 build.
Yes, it supports nested records.
On Fri, Oct 16, 2015 at 8:28 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
>
> Should it be supporting Records as a data type? (use to map sub
> structures).
>
> Regards,
> -Stefán
>
>
>
>
> On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com> wrote:
>
> > Hi Stefan,
> > We have enhanced Drill Avro Reader by supporting complex data types like
> > Union, Map etc.. This is targeted for 1.2.0 release. This release will
> > happen in a day or in a couple days.
> >
> > On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <
> stefan@activitystream.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm changing our logging to use the Avro format and there I'm relying
> on
> > > Maps and Records (f.ex. union ["null","RecordType"]) in addition to
> > > primitive types.
> > >
> > > I'm getting this error in Drill when I try to query the test file:
> > >
> > > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type
> must
> > be
> > > of the format : ["null", "some-type"]
> > >
> > >
> > > This error message provides no clues to which field are involved here
> but
> > > I'm hoping that this does not mean that sub structures, available in
> both
> > > JSON and Parquet, are not available in Avro as Records.
> > >
> > > Can someone please shed some light on this.
> > >
> > > Regards,
> > > -Stefan
> > >
> >
> >
> >
> > --
> > Kamesh.
> >
>
--
Kamesh.
Re: Avro support
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
Sorry I forgot to mention that I'm already using a 1.2-SNAPSHOT
Should it be supporting Records as a data type? (use to map sub structures).
Regards,
-Stefán
On Fri, Oct 16, 2015 at 2:31 PM, Kamesh <ka...@gmail.com> wrote:
> Hi Stefan,
> We have enhanced Drill Avro Reader by supporting complex data types like
> Union, Map etc.. This is targeted for 1.2.0 release. This release will
> happen in a day or in a couple days.
>
> On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > I'm changing our logging to use the Avro format and there I'm relying on
> > Maps and Records (f.ex. union ["null","RecordType"]) in addition to
> > primitive types.
> >
> > I'm getting this error in Drill when I try to query the test file:
> >
> > Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type must
> be
> > of the format : ["null", "some-type"]
> >
> >
> > This error message provides no clues to which field are involved here but
> > I'm hoping that this does not mean that sub structures, available in both
> > JSON and Parquet, are not available in Avro as Records.
> >
> > Can someone please shed some light on this.
> >
> > Regards,
> > -Stefan
> >
>
>
>
> --
> Kamesh.
>
Re: Avro support
Posted by Kamesh <ka...@gmail.com>.
Hi Stefan,
We have enhanced Drill Avro Reader by supporting complex data types like
Union, Map etc.. This is targeted for 1.2.0 release. This release will
happen in a day or in a couple days.
On Fri, Oct 16, 2015 at 6:30 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> I'm changing our logging to use the Avro format and there I'm relying on
> Maps and Records (f.ex. union ["null","RecordType"]) in addition to
> primitive types.
>
> I'm getting this error in Drill when I try to query the test file:
>
> Error: SYSTEM ERROR: UnsupportedOperationException: Avro union type must be
> of the format : ["null", "some-type"]
>
>
> This error message provides no clues to which field are involved here but
> I'm hoping that this does not mean that sub structures, available in both
> JSON and Parquet, are not available in Avro as Records.
>
> Can someone please shed some light on this.
>
> Regards,
> -Stefan
>
--
Kamesh.