You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Kim <da...@braintreepayments.com> on 2016/03/08 19:12:09 UTC
Stack overflow from self referencing Avro schema
Hello all,
I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
schema that has a self reference. For example:
item.avsc
{
"namespace": "..."
"type": "record"
"name": "Item",
"fields": [
{
"name": "parent"
"type": ["null, "Item"]
}
]
}
When running my flink job, I'm running into the follow error:
Exception in thread "Thread-94" java.lang.StackOverflowError
at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
Interestingly if I change the type to an Avro array in the schema, this
error is not thrown.
Thanks!
David
Re: Stack overflow from self referencing Avro schema
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
Please try to reproduce the problem in a simple (commandline) Java
application (i.e. without Flink and such, just Avro).
If you can reproduce it with Avro 1.8.0 then please file a bug report
(preferable with the simplest reproduction path you can come up with) via.
https://issues.apache.org/jira/browse/AVRO/
Thanks
Niels Basjes
On Thu, Mar 10, 2016 at 4:11 PM, David Kim <da...@braintreepayments.com>
wrote:
> Hello!
>
> Just wanted to check up on this again. Has anyone else seen this before or
> have any suggestions?
>
> Thanks!
> David
>
> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Hello all,
>>
>> I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
>> schema that has a self reference. For example:
>>
>> item.avsc
>>
>> {
>>
>> "namespace": "..."
>>
>> "type": "record"
>> "name": "Item",
>> "fields": [
>> {
>> "name": "parent"
>> "type": ["null, "Item"]
>> }
>> ]
>> }
>>
>>
>> When running my flink job, I'm running into the follow error:
>>
>> Exception in thread "Thread-94" java.lang.StackOverflowError
>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>
>>
>> Interestingly if I change the type to an Avro array in the schema, this
>> error is not thrown.
>>
>> Thanks!
>> David
>>
>
>
>
> --
> Note: this information is confidential. It is prohibited to share, post
> online or otherwise publicize without Braintree's prior written consent.
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Stack overflow from self referencing Avro schema
Posted by David Kim <da...@braintreepayments.com>.
Hi Robert!
Thank you! :)
David
On Tue, Mar 22, 2016 at 7:59 AM, Robert Metzger <rm...@apache.org> wrote:
> Hey David,
>
> FLINK-3602 has been merged to master.
>
> On Fri, Mar 11, 2016 at 5:11 PM, David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Thanks Stephan! :)
>>
>> On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> The following issue should track that.
>>> https://issues.apache.org/jira/browse/FLINK-3602
>>>
>>> @Niels: Thanks for looking into this. At this point, I think it may
>>> actually be a Flink issue, since it concerns the interaction of Avro and
>>> Flink's TypeInformation.
>>>
>>> On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Hi!
>>>>
>>>> I think that is a TypeExtractor bug. It may actually be a bug for all
>>>> recursive types.
>>>> Let's check this and come up with a fix...
>>>>
>>>> Greetings,
>>>> Stephan
>>>>
>>>>
>>>> On Thu, Mar 10, 2016 at 4:11 PM, David Kim <
>>>> david.kim@braintreepayments.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> Just wanted to check up on this again. Has anyone else seen this
>>>>> before or have any suggestions?
>>>>>
>>>>> Thanks!
>>>>> David
>>>>>
>>>>> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
>>>>> david.kim@braintreepayments.com> wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> I'm running into a StackOverflowError using flink 1.0.0. I have an
>>>>>> Avro schema that has a self reference. For example:
>>>>>>
>>>>>> item.avsc
>>>>>>
>>>>>> {
>>>>>>
>>>>>> "namespace": "..."
>>>>>>
>>>>>> "type": "record"
>>>>>> "name": "Item",
>>>>>> "fields": [
>>>>>> {
>>>>>> "name": "parent"
>>>>>> "type": ["null, "Item"]
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>>
>>>>>>
>>>>>> When running my flink job, I'm running into the follow error:
>>>>>>
>>>>>> Exception in thread "Thread-94" java.lang.StackOverflowError
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>>>
>>>>>>
>>>>>> Interestingly if I change the type to an Avro array in the schema,
>>>>>> this error is not thrown.
>>>>>>
>>>>>> Thanks!
>>>>>> David
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Note: this information is confidential. It is prohibited to share,
>>>>> post online or otherwise publicize without Braintree's prior written
>>>>> consent.
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Note: this information is confidential. It is prohibited to share, post
>> online or otherwise publicize without Braintree's prior written consent.
>>
>
>
--
Note: this information is confidential. It is prohibited to share, post
online or otherwise publicize without Braintree's prior written consent.
Re: Stack overflow from self referencing Avro schema
Posted by Robert Metzger <rm...@apache.org>.
Hey David,
FLINK-3602 has been merged to master.
On Fri, Mar 11, 2016 at 5:11 PM, David Kim <da...@braintreepayments.com>
wrote:
> Thanks Stephan! :)
>
> On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> The following issue should track that.
>> https://issues.apache.org/jira/browse/FLINK-3602
>>
>> @Niels: Thanks for looking into this. At this point, I think it may
>> actually be a Flink issue, since it concerns the interaction of Avro and
>> Flink's TypeInformation.
>>
>> On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi!
>>>
>>> I think that is a TypeExtractor bug. It may actually be a bug for all
>>> recursive types.
>>> Let's check this and come up with a fix...
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>>> On Thu, Mar 10, 2016 at 4:11 PM, David Kim <
>>> david.kim@braintreepayments.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> Just wanted to check up on this again. Has anyone else seen this before
>>>> or have any suggestions?
>>>>
>>>> Thanks!
>>>> David
>>>>
>>>> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
>>>> david.kim@braintreepayments.com> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I'm running into a StackOverflowError using flink 1.0.0. I have an
>>>>> Avro schema that has a self reference. For example:
>>>>>
>>>>> item.avsc
>>>>>
>>>>> {
>>>>>
>>>>> "namespace": "..."
>>>>>
>>>>> "type": "record"
>>>>> "name": "Item",
>>>>> "fields": [
>>>>> {
>>>>> "name": "parent"
>>>>> "type": ["null, "Item"]
>>>>> }
>>>>> ]
>>>>> }
>>>>>
>>>>>
>>>>> When running my flink job, I'm running into the follow error:
>>>>>
>>>>> Exception in thread "Thread-94" java.lang.StackOverflowError
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>>
>>>>>
>>>>> Interestingly if I change the type to an Avro array in the schema,
>>>>> this error is not thrown.
>>>>>
>>>>> Thanks!
>>>>> David
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Note: this information is confidential. It is prohibited to share, post
>>>> online or otherwise publicize without Braintree's prior written consent.
>>>>
>>>
>>>
>>
>
>
> --
> Note: this information is confidential. It is prohibited to share, post
> online or otherwise publicize without Braintree's prior written consent.
>
Re: Stack overflow from self referencing Avro schema
Posted by David Kim <da...@braintreepayments.com>.
Thanks Stephan! :)
On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <se...@apache.org> wrote:
> The following issue should track that.
> https://issues.apache.org/jira/browse/FLINK-3602
>
> @Niels: Thanks for looking into this. At this point, I think it may
> actually be a Flink issue, since it concerns the interaction of Avro and
> Flink's TypeInformation.
>
> On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> I think that is a TypeExtractor bug. It may actually be a bug for all
>> recursive types.
>> Let's check this and come up with a fix...
>>
>> Greetings,
>> Stephan
>>
>>
>> On Thu, Mar 10, 2016 at 4:11 PM, David Kim <
>> david.kim@braintreepayments.com> wrote:
>>
>>> Hello!
>>>
>>> Just wanted to check up on this again. Has anyone else seen this before
>>> or have any suggestions?
>>>
>>> Thanks!
>>> David
>>>
>>> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
>>> david.kim@braintreepayments.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
>>>> schema that has a self reference. For example:
>>>>
>>>> item.avsc
>>>>
>>>> {
>>>>
>>>> "namespace": "..."
>>>>
>>>> "type": "record"
>>>> "name": "Item",
>>>> "fields": [
>>>> {
>>>> "name": "parent"
>>>> "type": ["null, "Item"]
>>>> }
>>>> ]
>>>> }
>>>>
>>>>
>>>> When running my flink job, I'm running into the follow error:
>>>>
>>>> Exception in thread "Thread-94" java.lang.StackOverflowError
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>>
>>>>
>>>> Interestingly if I change the type to an Avro array in the schema,
>>>> this error is not thrown.
>>>>
>>>> Thanks!
>>>> David
>>>>
>>>
>>>
>>>
>>> --
>>> Note: this information is confidential. It is prohibited to share, post
>>> online or otherwise publicize without Braintree's prior written consent.
>>>
>>
>>
>
--
Note: this information is confidential. It is prohibited to share, post
online or otherwise publicize without Braintree's prior written consent.
Re: Stack overflow from self referencing Avro schema
Posted by Stephan Ewen <se...@apache.org>.
The following issue should track that.
https://issues.apache.org/jira/browse/FLINK-3602
@Niels: Thanks for looking into this. At this point, I think it may
actually be a Flink issue, since it concerns the interaction of Avro and
Flink's TypeInformation.
On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> I think that is a TypeExtractor bug. It may actually be a bug for all
> recursive types.
> Let's check this and come up with a fix...
>
> Greetings,
> Stephan
>
>
> On Thu, Mar 10, 2016 at 4:11 PM, David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Hello!
>>
>> Just wanted to check up on this again. Has anyone else seen this before
>> or have any suggestions?
>>
>> Thanks!
>> David
>>
>> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
>> david.kim@braintreepayments.com> wrote:
>>
>>> Hello all,
>>>
>>> I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
>>> schema that has a self reference. For example:
>>>
>>> item.avsc
>>>
>>> {
>>>
>>> "namespace": "..."
>>>
>>> "type": "record"
>>> "name": "Item",
>>> "fields": [
>>> {
>>> "name": "parent"
>>> "type": ["null, "Item"]
>>> }
>>> ]
>>> }
>>>
>>>
>>> When running my flink job, I'm running into the follow error:
>>>
>>> Exception in thread "Thread-94" java.lang.StackOverflowError
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>>
>>>
>>> Interestingly if I change the type to an Avro array in the schema, this
>>> error is not thrown.
>>>
>>> Thanks!
>>> David
>>>
>>
>>
>>
>> --
>> Note: this information is confidential. It is prohibited to share, post
>> online or otherwise publicize without Braintree's prior written consent.
>>
>
>
Re: Stack overflow from self referencing Avro schema
Posted by Stephan Ewen <se...@apache.org>.
Hi!
I think that is a TypeExtractor bug. It may actually be a bug for all
recursive types.
Let's check this and come up with a fix...
Greetings,
Stephan
On Thu, Mar 10, 2016 at 4:11 PM, David Kim <da...@braintreepayments.com>
wrote:
> Hello!
>
> Just wanted to check up on this again. Has anyone else seen this before or
> have any suggestions?
>
> Thanks!
> David
>
> On Tue, Mar 8, 2016 at 12:12 PM, David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Hello all,
>>
>> I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
>> schema that has a self reference. For example:
>>
>> item.avsc
>>
>> {
>>
>> "namespace": "..."
>>
>> "type": "record"
>> "name": "Item",
>> "fields": [
>> {
>> "name": "parent"
>> "type": ["null, "Item"]
>> }
>> ]
>> }
>>
>>
>> When running my flink job, I'm running into the follow error:
>>
>> Exception in thread "Thread-94" java.lang.StackOverflowError
>> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
>> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
>> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>>
>>
>> Interestingly if I change the type to an Avro array in the schema, this
>> error is not thrown.
>>
>> Thanks!
>> David
>>
>
>
>
> --
> Note: this information is confidential. It is prohibited to share, post
> online or otherwise publicize without Braintree's prior written consent.
>
Re: Stack overflow from self referencing Avro schema
Posted by David Kim <da...@braintreepayments.com>.
Hello!
Just wanted to check up on this again. Has anyone else seen this before or
have any suggestions?
Thanks!
David
On Tue, Mar 8, 2016 at 12:12 PM, David Kim <da...@braintreepayments.com>
wrote:
> Hello all,
>
> I'm running into a StackOverflowError using flink 1.0.0. I have an Avro
> schema that has a self reference. For example:
>
> item.avsc
>
> {
>
> "namespace": "..."
>
> "type": "record"
> "name": "Item",
> "fields": [
> {
> "name": "parent"
> "type": ["null, "Item"]
> }
> ]
> }
>
>
> When running my flink job, I'm running into the follow error:
>
> Exception in thread "Thread-94" java.lang.StackOverflowError
> at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
> at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
> at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
> at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
> at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
> at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
>
>
> Interestingly if I change the type to an Avro array in the schema, this
> error is not thrown.
>
> Thanks!
> David
>
--
Note: this information is confidential. It is prohibited to share, post
online or otherwise publicize without Braintree's prior written consent.