You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Van Boxel <al...@vanboxel.be> on 2019/08/17 14:20:54 UTC
Is it too late to switch to Java 8 time for the schema aware Row and
Beam SQL?
I know it's probably futile, but the more I'm working on features that are
related to schema awareness I'm getting a bit frustrated about the lack of
precision of the joda instance.
As soon as we have a conversion to the DateTime I need to drop precession,
this happens with the Protobuf timestamp (nanoseconds), but I also notice
it with BigQuery (milliseconds).
Suggestions?
_/
_/ Alex Van Boxel
Re: Is it too late to switch to Java 8 time for the schema aware Row
and Beam SQL?
Posted by Rui Wang <ru...@google.com>.
We need to figure out a way to make sure we can read the data without
losing precision. It will likely be case by case.
-Rui
On Wed, Aug 28, 2019 at 2:27 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> Thanks, how will ZetaSQL support higher precision as the input in general
> will be Instant anyway. Will it rely on the "pending" standardized logical
> types?
>
> _/
> _/ Alex Van Boxel
>
>
> On Mon, Aug 19, 2019 at 7:02 AM Rui Wang <ru...@google.com> wrote:
>
>> However, more challengings come from:
>>
>> 1. How to read data without losing precision. Beam Java SDK uses Joda
>> already so it's very likely that you will need update IO somehow to support
>> higher precision.
>> 2. How to process higher precision in BeamSQL. It means SQL functions
>> should support higher precision. If you use Beam Calcite, unfortunately it
>> will only support up to millis. If you use Beam ZetaSQL (under review),
>> there are opportunities to support higher precision for SQL functions.
>>
>>
>> -Rui
>>
>> On Sun, Aug 18, 2019 at 9:52 PM Rui Wang <ru...@google.com> wrote:
>>
>>> We have been discussing it for a long time. I think if you only want to
>>> support more precision (e.g. up to nanosecond) for BeamSQL, it's actually
>>> relatively straightforward to support it by using a logical type for
>>> BeamSQL.
>>>
>>>
>>> -Rui
>>>
>>> On Sat, Aug 17, 2019 at 7:21 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>>> I know it's probably futile, but the more I'm working on features that
>>>> are related to schema awareness I'm getting a bit frustrated about the lack
>>>> of precision of the joda instance.
>>>>
>>>> As soon as we have a conversion to the DateTime I need to drop
>>>> precession, this happens with the Protobuf timestamp (nanoseconds), but I
>>>> also notice it with BigQuery (milliseconds).
>>>>
>>>> Suggestions?
>>>>
>>>> _/
>>>> _/ Alex Van Boxel
>>>>
>>>
Re: Is it too late to switch to Java 8 time for the schema aware Row
and Beam SQL?
Posted by Alex Van Boxel <al...@vanboxel.be>.
Thanks, how will ZetaSQL support higher precision as the input in general
will be Instant anyway. Will it rely on the "pending" standardized logical
types?
_/
_/ Alex Van Boxel
On Mon, Aug 19, 2019 at 7:02 AM Rui Wang <ru...@google.com> wrote:
> However, more challengings come from:
>
> 1. How to read data without losing precision. Beam Java SDK uses Joda
> already so it's very likely that you will need update IO somehow to support
> higher precision.
> 2. How to process higher precision in BeamSQL. It means SQL functions
> should support higher precision. If you use Beam Calcite, unfortunately it
> will only support up to millis. If you use Beam ZetaSQL (under review),
> there are opportunities to support higher precision for SQL functions.
>
>
> -Rui
>
> On Sun, Aug 18, 2019 at 9:52 PM Rui Wang <ru...@google.com> wrote:
>
>> We have been discussing it for a long time. I think if you only want to
>> support more precision (e.g. up to nanosecond) for BeamSQL, it's actually
>> relatively straightforward to support it by using a logical type for
>> BeamSQL.
>>
>>
>> -Rui
>>
>> On Sat, Aug 17, 2019 at 7:21 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>
>>> I know it's probably futile, but the more I'm working on features that
>>> are related to schema awareness I'm getting a bit frustrated about the lack
>>> of precision of the joda instance.
>>>
>>> As soon as we have a conversion to the DateTime I need to drop
>>> precession, this happens with the Protobuf timestamp (nanoseconds), but I
>>> also notice it with BigQuery (milliseconds).
>>>
>>> Suggestions?
>>>
>>> _/
>>> _/ Alex Van Boxel
>>>
>>
Re: Is it too late to switch to Java 8 time for the schema aware Row
and Beam SQL?
Posted by Rui Wang <ru...@google.com>.
However, more challengings come from:
1. How to read data without losing precision. Beam Java SDK uses Joda
already so it's very likely that you will need update IO somehow to support
higher precision.
2. How to process higher precision in BeamSQL. It means SQL functions
should support higher precision. If you use Beam Calcite, unfortunately it
will only support up to millis. If you use Beam ZetaSQL (under review),
there are opportunities to support higher precision for SQL functions.
-Rui
On Sun, Aug 18, 2019 at 9:52 PM Rui Wang <ru...@google.com> wrote:
> We have been discussing it for a long time. I think if you only want to
> support more precision (e.g. up to nanosecond) for BeamSQL, it's actually
> relatively straightforward to support it by using a logical type for
> BeamSQL.
>
>
> -Rui
>
> On Sat, Aug 17, 2019 at 7:21 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>
>> I know it's probably futile, but the more I'm working on features that
>> are related to schema awareness I'm getting a bit frustrated about the lack
>> of precision of the joda instance.
>>
>> As soon as we have a conversion to the DateTime I need to drop
>> precession, this happens with the Protobuf timestamp (nanoseconds), but I
>> also notice it with BigQuery (milliseconds).
>>
>> Suggestions?
>>
>> _/
>> _/ Alex Van Boxel
>>
>
Re: Is it too late to switch to Java 8 time for the schema aware Row
and Beam SQL?
Posted by Rui Wang <ru...@google.com>.
We have been discussing it for a long time. I think if you only want to
support more precision (e.g. up to nanosecond) for BeamSQL, it's actually
relatively straightforward to support it by using a logical type for
BeamSQL.
-Rui
On Sat, Aug 17, 2019 at 7:21 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> I know it's probably futile, but the more I'm working on features that are
> related to schema awareness I'm getting a bit frustrated about the lack of
> precision of the joda instance.
>
> As soon as we have a conversion to the DateTime I need to drop precession,
> this happens with the Protobuf timestamp (nanoseconds), but I also notice
> it with BigQuery (milliseconds).
>
> Suggestions?
>
> _/
> _/ Alex Van Boxel
>