You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Brian Hulette <bh...@google.com> on 2021/09/03 18:15:28 UTC
Re: Dropping @Experimental from SqlTransform
Note Schemas are also Experimental and are used by SqlTransform. Should we
also drop the Experimental tag for schemas? I can't decide, so I'll instead
present my internal debate:
Argument against dropping Experimental:
There are some controversial parts of the schemas type system that we may
want to solidify (e.g. can we drop the primitive DATETIME type that is
actually an instant? we should agree on the definition of a logical type,
and define how they are distinct from a user type). It would be nice to
address these before dropping Experimental.
Argument for dropping Experimental:
Schemas are likely used widely enough that any breaking changes should be
announced and go through a deprecation cycle, which is inconsistent with
the @Experimental tag.
Brian
On Wed, Aug 4, 2021 at 3:01 PM Andrew Pilloud <ap...@google.com> wrote:
> Thank you for the feedback everyone. I've updated the PR to keep the UDF
> interfaces Experimental for now based on feedback directly on the PR. I
> plan to merge once those flaky tests are passing!
>
> Andrew
>
> On Mon, Aug 2, 2021 at 8:20 AM Tyson Hamilton <ty...@google.com> wrote:
>
>> +1 sounds good! Thanks Andrew.
>>
>> On Thu, Jul 29, 2021 at 2:46 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> Makes sense to me.
>>>
>>> On Thu, Jul 29, 2021 at 11:12 AM Andrew Pilloud <ap...@google.com>
>>> wrote:
>>> >
>>> > Hi everyone,
>>> >
>>> > Beam's Calcite SqlTransform has been stable for the last few years.
>>> We've only been adding new methods and I don't expect us to make any
>>> breaking changes. I believe it is time we drop the @Experimental
>>> annotation. (I am not proposing we drop it from ZetaSQL.)
>>> >
>>> > I've opened a PR (where I've also fixed the spotless issues):
>>> https://github.com/apache/beam/pull/15244
>>> >
>>> > Andrew
>>>
>>
Re: Dropping @Experimental from SqlTransform
Posted by Robert Bradshaw <ro...@google.com>.
Another alternative would be to drop Experimental on Schemas, but keep
it for some of the types that we're still not sure about, like
timestamps.
On Fri, Sep 3, 2021 at 11:18 AM Reuven Lax <re...@google.com> wrote:
>
> I think we should drop Experimental.
>
> For schemas, I would like to at least rename DATETIME to TIMESTAMP first. There's an argument for going further (removing DATETIME and making it a logical type), but that has been tricky when we've tried it in the past and would be harder for users to adapt to. I vote that we compromise by simply renaming the type and then removing Experimental.
>
> On Fri, Sep 3, 2021 at 11:15 AM Brian Hulette <bh...@google.com> wrote:
>>
>> Note Schemas are also Experimental and are used by SqlTransform. Should we also drop the Experimental tag for schemas? I can't decide, so I'll instead present my internal debate:
>>
>> Argument against dropping Experimental:
>> There are some controversial parts of the schemas type system that we may want to solidify (e.g. can we drop the primitive DATETIME type that is actually an instant? we should agree on the definition of a logical type, and define how they are distinct from a user type). It would be nice to address these before dropping Experimental.
>>
>> Argument for dropping Experimental:
>> Schemas are likely used widely enough that any breaking changes should be announced and go through a deprecation cycle, which is inconsistent with the @Experimental tag.
>>
>> Brian
>>
>> On Wed, Aug 4, 2021 at 3:01 PM Andrew Pilloud <ap...@google.com> wrote:
>>>
>>> Thank you for the feedback everyone. I've updated the PR to keep the UDF interfaces Experimental for now based on feedback directly on the PR. I plan to merge once those flaky tests are passing!
>>>
>>> Andrew
>>>
>>> On Mon, Aug 2, 2021 at 8:20 AM Tyson Hamilton <ty...@google.com> wrote:
>>>>
>>>> +1 sounds good! Thanks Andrew.
>>>>
>>>> On Thu, Jul 29, 2021 at 2:46 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>
>>>>> Makes sense to me.
>>>>>
>>>>> On Thu, Jul 29, 2021 at 11:12 AM Andrew Pilloud <ap...@google.com> wrote:
>>>>> >
>>>>> > Hi everyone,
>>>>> >
>>>>> > Beam's Calcite SqlTransform has been stable for the last few years. We've only been adding new methods and I don't expect us to make any breaking changes. I believe it is time we drop the @Experimental annotation. (I am not proposing we drop it from ZetaSQL.)
>>>>> >
>>>>> > I've opened a PR (where I've also fixed the spotless issues): https://github.com/apache/beam/pull/15244
>>>>> >
>>>>> > Andrew
Re: Dropping @Experimental from SqlTransform
Posted by Reuven Lax <re...@google.com>.
I think we should drop Experimental.
For schemas, I would like to at least rename DATETIME to TIMESTAMP first.
There's an argument for going further (removing DATETIME and making it a
logical type), but that has been tricky when we've tried it in the past and
would be harder for users to adapt to. I vote that we compromise by simply
renaming the type and then removing Experimental.
On Fri, Sep 3, 2021 at 11:15 AM Brian Hulette <bh...@google.com> wrote:
> Note Schemas are also Experimental and are used by SqlTransform. Should we
> also drop the Experimental tag for schemas? I can't decide, so I'll instead
> present my internal debate:
>
> Argument against dropping Experimental:
> There are some controversial parts of the schemas type system that we may
> want to solidify (e.g. can we drop the primitive DATETIME type that is
> actually an instant? we should agree on the definition of a logical type,
> and define how they are distinct from a user type). It would be nice to
> address these before dropping Experimental.
>
> Argument for dropping Experimental:
> Schemas are likely used widely enough that any breaking changes should be
> announced and go through a deprecation cycle, which is inconsistent with
> the @Experimental tag.
>
> Brian
>
> On Wed, Aug 4, 2021 at 3:01 PM Andrew Pilloud <ap...@google.com> wrote:
>
>> Thank you for the feedback everyone. I've updated the PR to keep the UDF
>> interfaces Experimental for now based on feedback directly on the PR. I
>> plan to merge once those flaky tests are passing!
>>
>> Andrew
>>
>> On Mon, Aug 2, 2021 at 8:20 AM Tyson Hamilton <ty...@google.com> wrote:
>>
>>> +1 sounds good! Thanks Andrew.
>>>
>>> On Thu, Jul 29, 2021 at 2:46 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> Makes sense to me.
>>>>
>>>> On Thu, Jul 29, 2021 at 11:12 AM Andrew Pilloud <ap...@google.com>
>>>> wrote:
>>>> >
>>>> > Hi everyone,
>>>> >
>>>> > Beam's Calcite SqlTransform has been stable for the last few years.
>>>> We've only been adding new methods and I don't expect us to make any
>>>> breaking changes. I believe it is time we drop the @Experimental
>>>> annotation. (I am not proposing we drop it from ZetaSQL.)
>>>> >
>>>> > I've opened a PR (where I've also fixed the spotless issues):
>>>> https://github.com/apache/beam/pull/15244
>>>> >
>>>> > Andrew
>>>>
>>>