You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jozef Vilcek <jo...@gmail.com> on 2021/12/12 12:21:17 UTC
Convert Row with nullable fields to Protobuf fails
I did notice that protobuf schema translator supports nullable for proto
fiels [1]. E.g. if I want to a nullable string, then in proto I can use
`google.protobuf.StringValue` and schema will look fine.
However, fromRow creator does not support this and throw exception if it is
presented with row instance with null value. It tries to pass this null to
a `StringValue` while proto way would be probably ignoring the set in that
case.
I can submit a failing test case and JIRA but want to check first if this
is supposed to be supported - having row with nulls handled and
creating instance of compatible protobuf.
Thanks,
Jozo
[1]
https://github.com/apache/beam/blob/master/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L106
Re: Convert Row with nullable fields to Protobuf fails
Posted by Brian Hulette <bh...@google.com>.
Hi Jozo,
Thanks for the report and sorry for not getting back to you sooner. This
definitely seems like a bug given the documentation you linked, would you
be willing to file a jira? Also could you share the stacktrace for the
error you're getting?
Brian
On Sat, Dec 18, 2021 at 12:27 PM Jozef Vilcek <jo...@gmail.com> wrote:
> Any feedback on this one please?
>
> On Mon, Dec 13, 2021 at 11:02 AM Jozef Vilcek <jo...@gmail.com>
> wrote:
>
>> Yes, field is marked as nullable. Here is a test case change to
>> illustrate it:
>>
>> https://github.com/JozoVilcek/beam/commit/5e1c6324868c2fd6145dd2348c7358fdc787ac38
>>
>> On Sun, Dec 12, 2021 at 7:28 PM Reuven Lax <re...@google.com> wrote:
>>
>>> Is the schema field marked as nullable?
>>>
>>> On Sun, Dec 12, 2021 at 4:21 AM Jozef Vilcek <jo...@gmail.com>
>>> wrote:
>>>
>>>> I did notice that protobuf schema translator supports nullable for
>>>> proto fiels [1]. E.g. if I want to a nullable string, then in proto I can
>>>> use `google.protobuf.StringValue` and schema will look fine.
>>>>
>>>> However, fromRow creator does not support this and throw exception if
>>>> it is presented with row instance with null value. It tries to pass this
>>>> null to a `StringValue` while proto way would be probably ignoring the set
>>>> in that case.
>>>>
>>>> I can submit a failing test case and JIRA but want to check first if
>>>> this is supposed to be supported - having row with nulls handled and
>>>> creating instance of compatible protobuf.
>>>>
>>>> Thanks,
>>>> Jozo
>>>>
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L106
>>>>
>>>
Re: Convert Row with nullable fields to Protobuf fails
Posted by Jozef Vilcek <jo...@gmail.com>.
Any feedback on this one please?
On Mon, Dec 13, 2021 at 11:02 AM Jozef Vilcek <jo...@gmail.com> wrote:
> Yes, field is marked as nullable. Here is a test case change to
> illustrate it:
>
> https://github.com/JozoVilcek/beam/commit/5e1c6324868c2fd6145dd2348c7358fdc787ac38
>
> On Sun, Dec 12, 2021 at 7:28 PM Reuven Lax <re...@google.com> wrote:
>
>> Is the schema field marked as nullable?
>>
>> On Sun, Dec 12, 2021 at 4:21 AM Jozef Vilcek <jo...@gmail.com>
>> wrote:
>>
>>> I did notice that protobuf schema translator supports nullable for proto
>>> fiels [1]. E.g. if I want to a nullable string, then in proto I can use
>>> `google.protobuf.StringValue` and schema will look fine.
>>>
>>> However, fromRow creator does not support this and throw exception if it
>>> is presented with row instance with null value. It tries to pass this null
>>> to a `StringValue` while proto way would be probably ignoring the set in
>>> that case.
>>>
>>> I can submit a failing test case and JIRA but want to check first if
>>> this is supposed to be supported - having row with nulls handled and
>>> creating instance of compatible protobuf.
>>>
>>> Thanks,
>>> Jozo
>>>
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L106
>>>
>>
Re: Convert Row with nullable fields to Protobuf fails
Posted by Jozef Vilcek <jo...@gmail.com>.
Yes, field is marked as nullable. Here is a test case change to
illustrate it:
https://github.com/JozoVilcek/beam/commit/5e1c6324868c2fd6145dd2348c7358fdc787ac38
On Sun, Dec 12, 2021 at 7:28 PM Reuven Lax <re...@google.com> wrote:
> Is the schema field marked as nullable?
>
> On Sun, Dec 12, 2021 at 4:21 AM Jozef Vilcek <jo...@gmail.com>
> wrote:
>
>> I did notice that protobuf schema translator supports nullable for proto
>> fiels [1]. E.g. if I want to a nullable string, then in proto I can use
>> `google.protobuf.StringValue` and schema will look fine.
>>
>> However, fromRow creator does not support this and throw exception if it
>> is presented with row instance with null value. It tries to pass this null
>> to a `StringValue` while proto way would be probably ignoring the set in
>> that case.
>>
>> I can submit a failing test case and JIRA but want to check first if this
>> is supposed to be supported - having row with nulls handled and
>> creating instance of compatible protobuf.
>>
>> Thanks,
>> Jozo
>>
>>
>> [1]
>> https://github.com/apache/beam/blob/master/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L106
>>
>
Re: Convert Row with nullable fields to Protobuf fails
Posted by Reuven Lax <re...@google.com>.
Is the schema field marked as nullable?
On Sun, Dec 12, 2021 at 4:21 AM Jozef Vilcek <jo...@gmail.com> wrote:
> I did notice that protobuf schema translator supports nullable for proto
> fiels [1]. E.g. if I want to a nullable string, then in proto I can use
> `google.protobuf.StringValue` and schema will look fine.
>
> However, fromRow creator does not support this and throw exception if it
> is presented with row instance with null value. It tries to pass this null
> to a `StringValue` while proto way would be probably ignoring the set in
> that case.
>
> I can submit a failing test case and JIRA but want to check first if this
> is supposed to be supported - having row with nulls handled and
> creating instance of compatible protobuf.
>
> Thanks,
> Jozo
>
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L106
>