You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jozef Vilcek <jo...@gmail.com> on 2020/02/21 18:38:20 UTC
Schema Convert transform fails on type metadata
Hi,
I am playing with Schemas in Beam and conversion between types. I am
experimenting with Convert transform to convert PCollection of POJOs to
Protobufs, but get a failure about schemas not being compatible.
The root cause is that FieldType are not passing `equivalent()` check
because of difference in metadata as protobuf add there a proto_number
https://github.com/apache/beam/blob/v2.19.0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L818
Is it correct that metadata should be strictly equal to pass field type
equivalence check, or not?
Thanks,
Jozef
Re: Schema Convert transform fails on type metadata
Posted by Jozef Vilcek <jo...@gmail.com>.
Will do! Thanks Alex.
On Sat, Feb 22, 2020 at 9:23 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> I've assigned it too you. If you create a PR you can add me as reviewer.
>
> _/
> _/ Alex Van Boxel
>
>
> On Sat, Feb 22, 2020 at 9:14 AM Jozef Vilcek <jo...@gmail.com>
> wrote:
>
>> I have filed https://issues.apache.org/jira/browse/BEAM-9360 for this.
>>
>> Seems like I am not able to self assign this, can I get rights to do that
>> or get someone assign JIRA to me please?
>>
>> On Sat, Feb 22, 2020 at 2:41 AM Reuven Lax <re...@google.com> wrote:
>>
>>> I think that it's incorrect - we shouldn't enforce that the metadata
>>> match.
>>>
>>> On Fri, Feb 21, 2020 at 10:38 AM Jozef Vilcek <jo...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am playing with Schemas in Beam and conversion between types. I am
>>>> experimenting with Convert transform to convert PCollection of POJOs to
>>>> Protobufs, but get a failure about schemas not being compatible.
>>>>
>>>> The root cause is that FieldType are not passing `equivalent()` check
>>>> because of difference in metadata as protobuf add there a proto_number
>>>>
>>>> https://github.com/apache/beam/blob/v2.19.0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L818
>>>>
>>>> Is it correct that metadata should be strictly equal to pass field type
>>>> equivalence check, or not?
>>>>
>>>> Thanks,
>>>> Jozef
>>>>
>>>
Re: Schema Convert transform fails on type metadata
Posted by Alex Van Boxel <al...@vanboxel.be>.
I've assigned it too you. If you create a PR you can add me as reviewer.
_/
_/ Alex Van Boxel
On Sat, Feb 22, 2020 at 9:14 AM Jozef Vilcek <jo...@gmail.com> wrote:
> I have filed https://issues.apache.org/jira/browse/BEAM-9360 for this.
>
> Seems like I am not able to self assign this, can I get rights to do that
> or get someone assign JIRA to me please?
>
> On Sat, Feb 22, 2020 at 2:41 AM Reuven Lax <re...@google.com> wrote:
>
>> I think that it's incorrect - we shouldn't enforce that the metadata
>> match.
>>
>> On Fri, Feb 21, 2020 at 10:38 AM Jozef Vilcek <jo...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am playing with Schemas in Beam and conversion between types. I am
>>> experimenting with Convert transform to convert PCollection of POJOs to
>>> Protobufs, but get a failure about schemas not being compatible.
>>>
>>> The root cause is that FieldType are not passing `equivalent()` check
>>> because of difference in metadata as protobuf add there a proto_number
>>>
>>> https://github.com/apache/beam/blob/v2.19.0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L818
>>>
>>> Is it correct that metadata should be strictly equal to pass field type
>>> equivalence check, or not?
>>>
>>> Thanks,
>>> Jozef
>>>
>>
Re: Schema Convert transform fails on type metadata
Posted by Jozef Vilcek <jo...@gmail.com>.
I have filed https://issues.apache.org/jira/browse/BEAM-9360 for this.
Seems like I am not able to self assign this, can I get rights to do that
or get someone assign JIRA to me please?
On Sat, Feb 22, 2020 at 2:41 AM Reuven Lax <re...@google.com> wrote:
> I think that it's incorrect - we shouldn't enforce that the metadata match.
>
> On Fri, Feb 21, 2020 at 10:38 AM Jozef Vilcek <jo...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am playing with Schemas in Beam and conversion between types. I am
>> experimenting with Convert transform to convert PCollection of POJOs to
>> Protobufs, but get a failure about schemas not being compatible.
>>
>> The root cause is that FieldType are not passing `equivalent()` check
>> because of difference in metadata as protobuf add there a proto_number
>>
>> https://github.com/apache/beam/blob/v2.19.0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L818
>>
>> Is it correct that metadata should be strictly equal to pass field type
>> equivalence check, or not?
>>
>> Thanks,
>> Jozef
>>
>
Re: Schema Convert transform fails on type metadata
Posted by Reuven Lax <re...@google.com>.
I think that it's incorrect - we shouldn't enforce that the metadata match.
On Fri, Feb 21, 2020 at 10:38 AM Jozef Vilcek <jo...@gmail.com> wrote:
> Hi,
>
> I am playing with Schemas in Beam and conversion between types. I am
> experimenting with Convert transform to convert PCollection of POJOs to
> Protobufs, but get a failure about schemas not being compatible.
>
> The root cause is that FieldType are not passing `equivalent()` check
> because of difference in metadata as protobuf add there a proto_number
>
> https://github.com/apache/beam/blob/v2.19.0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L818
>
> Is it correct that metadata should be strictly equal to pass field type
> equivalence check, or not?
>
> Thanks,
> Jozef
>