You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Niel Markwick <ni...@google.com> on 2022/04/08 13:57:40 UTC

Converting a Java native type to a Beam Row

While looking at a problem relating to a Python pipeline that uses (Java)
SpannerIO, I came across the code
<https://github.com/apache/beam/blob/5b0e92f691486c034d6d4f4b315d3a7a7fe66cf5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/StructUtils.java#L290>
that converts a Spanner Java native type for returning rows from the
database (Struct) to a Beam Row.

There were a couple of things that jumped out at me that worried me

   1. the FieldTypes of the Beam Schema are ignored, and the Spanner types
   are used to determine which types to use, which may lead to unexpected
   classes in the Beam Row, and class cast exceptions (the case of the problem
   I found).
   2. Logical types are ignored (there are a few TODOs)
   3. Loss-of-precision (nanosecond timestamps truncated to seconds)
   4. Not all Spanner types are handled (normal as more types have been
   added over time).


I have not yet worked with Beam Schemas and Rows, so I am wondering if
these issues are normal...

It feels like not verifying that the FieldTypes of the Schema match the
object class being added to the Row makes the code somewhat fragile...

Am I overreacting here - or should I create a bug...
(I fear also that fixing some of these issues may create
backwards-incompatible changes)


-- 
<https://cloud.google.com> * •  **Niel Markwick*
* •  *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
* •  *Google Belgium
* •  *nielm@google.com


Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie.
RPR: 0878.065.378

If you have received this communication by mistake, please don't forward it
to anyone else (it may contain confidential or privileged information),
please erase all copies of it, including all attachments, and please let
the sender know it went to the wrong person. Thanks

Re: Converting a Java native type to a Beam Row

Posted by Brian Hulette <bh...@google.com>.
Yeah I agree all of these are concerning to varying degrees. I'd also like
to point out Python has an experimental native Spanner IO [1] that was
added before this xlang one.

> 1. the FieldTypes of the Beam Schema are ignored

In a Beam source the FieldTypes might be generated from the source's schema
at construction time. Could that be what's happening here? It's certainly
possible there's a mismatch between the schema-translation
(construction-time) and row-translation (execution-time) logic, though.

> 2. Logical types are ignored (there are a few TODOs), 4. Not all Spanner
types are handled

It's common to only handle certain logical types and reject others. This
should be a construction-time error (if not it's a bug).

> 3. Loss-of-precision (nanosecond timestamps truncated to seconds)

This is definitely a bug. I'm embarrassed I didn't catch this when
reviewing the PR [1] that added it. In some other places we require a user
to configure behavior for high-precision timestamps (always truncate, or
raise execution time error if a value is too precise) - I think that's what
we should do here. I'm not sure how to fix this in a backwards-compatible
way though. If we make "always truncate" the default to make it
backwards-compatible we continue to opt users in to losing precision.

Brian

[1]
https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/experimental/spannerio.py
[2] https://github.com/apache/beam/pull/12611


On Fri, Apr 8, 2022 at 6:58 AM Niel Markwick <ni...@google.com> wrote:

>
> While looking at a problem relating to a Python pipeline that uses (Java)
> SpannerIO, I came across the code
> <https://github.com/apache/beam/blob/5b0e92f691486c034d6d4f4b315d3a7a7fe66cf5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/StructUtils.java#L290>
> that converts a Spanner Java native type for returning rows from the
> database (Struct) to a Beam Row.
>
> There were a couple of things that jumped out at me that worried me
>
>    1. the FieldTypes of the Beam Schema are ignored, and the Spanner
>    types are used to determine which types to use, which may lead to
>    unexpected classes in the Beam Row, and class cast exceptions (the case of
>    the problem I found).
>    2. Logical types are ignored (there are a few TODOs)
>    3. Loss-of-precision (nanosecond timestamps truncated to seconds)
>    4. Not all Spanner types are handled (normal as more types have been
>    added over time).
>
>
> I have not yet worked with Beam Schemas and Rows, so I am wondering if
> these issues are normal...
>
> It feels like not verifying that the FieldTypes of the Schema match the
> object class being added to the Row makes the code somewhat fragile...
>
> Am I overreacting here - or should I create a bug...
> (I fear also that fixing some of these issues may create
> backwards-incompatible changes)
>
>
> --
> <https://cloud.google.com> * •  **Niel Markwick*
> * •  *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
> * •  *Google Belgium
> * •  *nielm@google.com
>
>
>
> Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: 0878.065.378
>
> If you have received this communication by mistake, please don't forward
> it to anyone else (it may contain confidential or privileged information),
> please erase all copies of it, including all attachments, and please let
> the sender know it went to the wrong person. Thanks
>