You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Yi Hu via dev <de...@beam.apache.org> on 2025/03/24 21:26:21 UTC
[Discussion] Deprecate ZetaSQL
Hi everyone,
I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
component [1].
Beam SQL currently serves with two SQL dialects (i) Apache Calcite and (ii)
ZetaSQL dialects, see documentation [2] due to the following reasons
- Developments in Beam for ZetaSQL dialect effectively stalled since early
2022 (See change history [3])
- Despite incomplete support status, there is no new bug / feature request
opened ever since we migrated to use GitHub Issue, suggesting minimal
adoption [4]
- We still need to keep zetasql up-to-date if its dependency conflicts with
other google dependencies, as a result ZetaSQL component introduces
maintenance burden when upgrading GCP-BOM (e.g. [5]).
- One of the main reason that using ZetaSQL dialect, per [2], was because
> ZetaSQL is more compatible with BigQuery, so it’s especially useful in
pipelines that write to or read from BigQuery tables.
As of today, as GCP BigQuery now supports using GoogleSQL (open-sourced
as ZetaSQL) querying data that's stored outside of BigQuery via BigQuery
Connections API / Federated query [6, 7]. This largely provides an
alternative for using Beam's ZetaSQL interacting with BigQuery.
For these reasons, I propose initiating the process of deprecating
Beam SQL's ZetaSQL component. There are two decisions needed to be made:
Firstly, agree on when to document the deprecated status for ZetaSQL
component in javadoc, beam website, currently I recommend do it in the
release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
2025)
Secondly, stop publishing ZetaSQL artifacts. This is a breaking change, and
I think we can leave the deprecated status as is until the following
situation emerges, whichever comes first, and no earlier than Beam 2.66.0
(cut Jun 11, 2025)
- Continued support for ZetaSQL component involving significant burdens,
like conflict with other Beam dependencies, supported Java versions, etc, or
- When Beam moved to the next release major release (3)
Thanks for your attention, and any input welcomed!
Regards,
Yi
[1]
https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
[2] https://beam.apache.org/documentation/dsls/sql/overview/
[3]
https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
[4]
https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
[5] https://github.com/apache/beam/pull/32902
[6] https://cloud.google.com/bigquery/docs/connections-api-intro
[7] https://cloud.google.com/bigquery/docs/federated-queries-intro
--
Yi Hu, (he/him/his)
Software Engineer
Re: [Discussion] Deprecate ZetaSQL
Posted by Yi Hu via user <us...@beam.apache.org>.
Thanks for the suggestion.
Both documentation and expansion time warnings are added in
https://github.com/apache/beam/pull/34563 .
Yi
On Mon, Mar 31, 2025 at 2:43 PM Kenneth Knowles <ke...@apache.org> wrote:
> One way that you might be able to reach some users is to issue a warning
> in the code for SqlTransform.expand(), with the deprecation timeline. Maybe
> this is under "etc" but I wanted to mention it because the other items were
> all documentation.
>
> Kenn
>
> On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <de...@beam.apache.org> wrote:
>
>> Thanks for inputs! From discussion we have agreed on moving forward
>>
>> Next step:
>>
>> - Draft PR to note the deprecation status in Documentation, including
>> CHANGES, Javadoc, etc
>>
>> - The earliest release to stop publishing ZetaSQL artifacts is pushed
>> further, not earlier than 1 full quarter before the next Release (2.65.0 +
>> 3 months would be 2.68.0), and only when it causes issues on maintenance.
>>
>> Thanks,
>>
>> Yi
>>
>>
>> On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> +1 to this deprecation. Thanks for putting together a clear summary.
>>>
>>> FWIW it also has significantly worse performance than Calcite SQL
>>> dialect, since it calls out to a ZetaSQL subprocess for most calculations,
>>> and that is less optimized than Beam's Fn API.
>>>
>>> Kenn
>>>
>>> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
>>> user@beam.apache.org> wrote:
>>>
>>>> I'm in favor of deprecating this and cleaning it up, but it depends on
>>>> usage. I suspect it is low (or possibly non-existent, especially as there's
>>>> little upside to moving away from the default). I cc'd user@ just in
>>>> case anyone wants to chime in there. This may be a good thing to add to our
>>>> release notes as well (perhaps we can get it in the one that's just about
>>>> to go out).
>>>>
>>>> Unless there is strong, justified pushback, I'd get the deprecation
>>>> status (e.g. on the javadocs, website) right away. For actual removal, I
>>>> agree with the idea of waiting until it actually causes issues or we move
>>>> to the next major beam release, though I might push back at 2.66 being a
>>>> bit too quick even if the first condition is hit before then and might give
>>>> people at least a quarter's notice.
>>>>
>>>> - Robert
>>>>
>>>>
>>>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>>>> component [1].
>>>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>>>
>>>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>>>> early 2022 (See change history [3])
>>>>>
>>>>> - Despite incomplete support status, there is no new bug / feature
>>>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>>>> minimal adoption [4]
>>>>>
>>>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>>>> with other google dependencies, as a result ZetaSQL component introduces
>>>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>>>
>>>>> - One of the main reason that using ZetaSQL dialect, per [2], was
>>>>> because
>>>>>
>>>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful
>>>>> in pipelines that write to or read from BigQuery tables.
>>>>>
>>>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>>>
>>>>> For these reasons, I propose initiating the process of deprecating
>>>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>>>
>>>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>>>> component in javadoc, beam website, currently I recommend do it in the
>>>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>>>> 2025)
>>>>>
>>>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking
>>>>> change, and I think we can leave the deprecated status as is until the
>>>>> following situation emerges, whichever comes first, and no earlier than
>>>>> Beam 2.66.0 (cut Jun 11, 2025)
>>>>>
>>>>> - Continued support for ZetaSQL component involving significant
>>>>> burdens, like conflict with other Beam dependencies, supported Java
>>>>> versions, etc, or
>>>>> - When Beam moved to the next release major release (3)
>>>>>
>>>>> Thanks for your attention, and any input welcomed!
>>>>>
>>>>> Regards,
>>>>> Yi
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>>>> [3]
>>>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>>>> [4]
>>>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>>>> [5] https://github.com/apache/beam/pull/32902
>>>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>>>
>>>>> --
>>>>>
>>>>> Yi Hu, (he/him/his)
>>>>>
>>>>> Software Engineer
>>>>>
>>>>>
>>>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Yi Hu via dev <de...@beam.apache.org>.
Thanks for the suggestion.
Both documentation and expansion time warnings are added in
https://github.com/apache/beam/pull/34563 .
Yi
On Mon, Mar 31, 2025 at 2:43 PM Kenneth Knowles <ke...@apache.org> wrote:
> One way that you might be able to reach some users is to issue a warning
> in the code for SqlTransform.expand(), with the deprecation timeline. Maybe
> this is under "etc" but I wanted to mention it because the other items were
> all documentation.
>
> Kenn
>
> On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <de...@beam.apache.org> wrote:
>
>> Thanks for inputs! From discussion we have agreed on moving forward
>>
>> Next step:
>>
>> - Draft PR to note the deprecation status in Documentation, including
>> CHANGES, Javadoc, etc
>>
>> - The earliest release to stop publishing ZetaSQL artifacts is pushed
>> further, not earlier than 1 full quarter before the next Release (2.65.0 +
>> 3 months would be 2.68.0), and only when it causes issues on maintenance.
>>
>> Thanks,
>>
>> Yi
>>
>>
>> On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> +1 to this deprecation. Thanks for putting together a clear summary.
>>>
>>> FWIW it also has significantly worse performance than Calcite SQL
>>> dialect, since it calls out to a ZetaSQL subprocess for most calculations,
>>> and that is less optimized than Beam's Fn API.
>>>
>>> Kenn
>>>
>>> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
>>> user@beam.apache.org> wrote:
>>>
>>>> I'm in favor of deprecating this and cleaning it up, but it depends on
>>>> usage. I suspect it is low (or possibly non-existent, especially as there's
>>>> little upside to moving away from the default). I cc'd user@ just in
>>>> case anyone wants to chime in there. This may be a good thing to add to our
>>>> release notes as well (perhaps we can get it in the one that's just about
>>>> to go out).
>>>>
>>>> Unless there is strong, justified pushback, I'd get the deprecation
>>>> status (e.g. on the javadocs, website) right away. For actual removal, I
>>>> agree with the idea of waiting until it actually causes issues or we move
>>>> to the next major beam release, though I might push back at 2.66 being a
>>>> bit too quick even if the first condition is hit before then and might give
>>>> people at least a quarter's notice.
>>>>
>>>> - Robert
>>>>
>>>>
>>>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>>>> component [1].
>>>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>>>
>>>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>>>> early 2022 (See change history [3])
>>>>>
>>>>> - Despite incomplete support status, there is no new bug / feature
>>>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>>>> minimal adoption [4]
>>>>>
>>>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>>>> with other google dependencies, as a result ZetaSQL component introduces
>>>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>>>
>>>>> - One of the main reason that using ZetaSQL dialect, per [2], was
>>>>> because
>>>>>
>>>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful
>>>>> in pipelines that write to or read from BigQuery tables.
>>>>>
>>>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>>>
>>>>> For these reasons, I propose initiating the process of deprecating
>>>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>>>
>>>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>>>> component in javadoc, beam website, currently I recommend do it in the
>>>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>>>> 2025)
>>>>>
>>>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking
>>>>> change, and I think we can leave the deprecated status as is until the
>>>>> following situation emerges, whichever comes first, and no earlier than
>>>>> Beam 2.66.0 (cut Jun 11, 2025)
>>>>>
>>>>> - Continued support for ZetaSQL component involving significant
>>>>> burdens, like conflict with other Beam dependencies, supported Java
>>>>> versions, etc, or
>>>>> - When Beam moved to the next release major release (3)
>>>>>
>>>>> Thanks for your attention, and any input welcomed!
>>>>>
>>>>> Regards,
>>>>> Yi
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>>>> [3]
>>>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>>>> [4]
>>>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>>>> [5] https://github.com/apache/beam/pull/32902
>>>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>>>
>>>>> --
>>>>>
>>>>> Yi Hu, (he/him/his)
>>>>>
>>>>> Software Engineer
>>>>>
>>>>>
>>>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Kenneth Knowles <ke...@apache.org>.
One way that you might be able to reach some users is to issue a warning in
the code for SqlTransform.expand(), with the deprecation timeline. Maybe
this is under "etc" but I wanted to mention it because the other items were
all documentation.
Kenn
On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <de...@beam.apache.org> wrote:
> Thanks for inputs! From discussion we have agreed on moving forward
>
> Next step:
>
> - Draft PR to note the deprecation status in Documentation, including
> CHANGES, Javadoc, etc
>
> - The earliest release to stop publishing ZetaSQL artifacts is pushed
> further, not earlier than 1 full quarter before the next Release (2.65.0 +
> 3 months would be 2.68.0), and only when it causes issues on maintenance.
>
> Thanks,
>
> Yi
>
>
> On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> +1 to this deprecation. Thanks for putting together a clear summary.
>>
>> FWIW it also has significantly worse performance than Calcite SQL
>> dialect, since it calls out to a ZetaSQL subprocess for most calculations,
>> and that is less optimized than Beam's Fn API.
>>
>> Kenn
>>
>> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
>> user@beam.apache.org> wrote:
>>
>>> I'm in favor of deprecating this and cleaning it up, but it depends on
>>> usage. I suspect it is low (or possibly non-existent, especially as there's
>>> little upside to moving away from the default). I cc'd user@ just in
>>> case anyone wants to chime in there. This may be a good thing to add to our
>>> release notes as well (perhaps we can get it in the one that's just about
>>> to go out).
>>>
>>> Unless there is strong, justified pushback, I'd get the deprecation
>>> status (e.g. on the javadocs, website) right away. For actual removal, I
>>> agree with the idea of waiting until it actually causes issues or we move
>>> to the next major beam release, though I might push back at 2.66 being a
>>> bit too quick even if the first condition is hit before then and might give
>>> people at least a quarter's notice.
>>>
>>> - Robert
>>>
>>>
>>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>>> component [1].
>>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>>
>>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>>> early 2022 (See change history [3])
>>>>
>>>> - Despite incomplete support status, there is no new bug / feature
>>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>>> minimal adoption [4]
>>>>
>>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>>> with other google dependencies, as a result ZetaSQL component introduces
>>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>>
>>>> - One of the main reason that using ZetaSQL dialect, per [2], was
>>>> because
>>>>
>>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful
>>>> in pipelines that write to or read from BigQuery tables.
>>>>
>>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>>
>>>> For these reasons, I propose initiating the process of deprecating
>>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>>
>>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>>> component in javadoc, beam website, currently I recommend do it in the
>>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>>> 2025)
>>>>
>>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>>>> and I think we can leave the deprecated status as is until the following
>>>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>>>> (cut Jun 11, 2025)
>>>>
>>>> - Continued support for ZetaSQL component involving significant
>>>> burdens, like conflict with other Beam dependencies, supported Java
>>>> versions, etc, or
>>>> - When Beam moved to the next release major release (3)
>>>>
>>>> Thanks for your attention, and any input welcomed!
>>>>
>>>> Regards,
>>>> Yi
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>>> [3]
>>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>>> [4]
>>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>>> [5] https://github.com/apache/beam/pull/32902
>>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>>
>>>> --
>>>>
>>>> Yi Hu, (he/him/his)
>>>>
>>>> Software Engineer
>>>>
>>>>
>>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Kenneth Knowles <ke...@apache.org>.
One way that you might be able to reach some users is to issue a warning in
the code for SqlTransform.expand(), with the deprecation timeline. Maybe
this is under "etc" but I wanted to mention it because the other items were
all documentation.
Kenn
On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <de...@beam.apache.org> wrote:
> Thanks for inputs! From discussion we have agreed on moving forward
>
> Next step:
>
> - Draft PR to note the deprecation status in Documentation, including
> CHANGES, Javadoc, etc
>
> - The earliest release to stop publishing ZetaSQL artifacts is pushed
> further, not earlier than 1 full quarter before the next Release (2.65.0 +
> 3 months would be 2.68.0), and only when it causes issues on maintenance.
>
> Thanks,
>
> Yi
>
>
> On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> +1 to this deprecation. Thanks for putting together a clear summary.
>>
>> FWIW it also has significantly worse performance than Calcite SQL
>> dialect, since it calls out to a ZetaSQL subprocess for most calculations,
>> and that is less optimized than Beam's Fn API.
>>
>> Kenn
>>
>> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
>> user@beam.apache.org> wrote:
>>
>>> I'm in favor of deprecating this and cleaning it up, but it depends on
>>> usage. I suspect it is low (or possibly non-existent, especially as there's
>>> little upside to moving away from the default). I cc'd user@ just in
>>> case anyone wants to chime in there. This may be a good thing to add to our
>>> release notes as well (perhaps we can get it in the one that's just about
>>> to go out).
>>>
>>> Unless there is strong, justified pushback, I'd get the deprecation
>>> status (e.g. on the javadocs, website) right away. For actual removal, I
>>> agree with the idea of waiting until it actually causes issues or we move
>>> to the next major beam release, though I might push back at 2.66 being a
>>> bit too quick even if the first condition is hit before then and might give
>>> people at least a quarter's notice.
>>>
>>> - Robert
>>>
>>>
>>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>>> component [1].
>>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>>
>>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>>> early 2022 (See change history [3])
>>>>
>>>> - Despite incomplete support status, there is no new bug / feature
>>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>>> minimal adoption [4]
>>>>
>>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>>> with other google dependencies, as a result ZetaSQL component introduces
>>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>>
>>>> - One of the main reason that using ZetaSQL dialect, per [2], was
>>>> because
>>>>
>>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful
>>>> in pipelines that write to or read from BigQuery tables.
>>>>
>>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>>
>>>> For these reasons, I propose initiating the process of deprecating
>>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>>
>>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>>> component in javadoc, beam website, currently I recommend do it in the
>>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>>> 2025)
>>>>
>>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>>>> and I think we can leave the deprecated status as is until the following
>>>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>>>> (cut Jun 11, 2025)
>>>>
>>>> - Continued support for ZetaSQL component involving significant
>>>> burdens, like conflict with other Beam dependencies, supported Java
>>>> versions, etc, or
>>>> - When Beam moved to the next release major release (3)
>>>>
>>>> Thanks for your attention, and any input welcomed!
>>>>
>>>> Regards,
>>>> Yi
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>>> [3]
>>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>>> [4]
>>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>>> [5] https://github.com/apache/beam/pull/32902
>>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>>
>>>> --
>>>>
>>>> Yi Hu, (he/him/his)
>>>>
>>>> Software Engineer
>>>>
>>>>
>>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Yi Hu via dev <de...@beam.apache.org>.
Thanks for inputs! From discussion we have agreed on moving forward
Next step:
- Draft PR to note the deprecation status in Documentation, including
CHANGES, Javadoc, etc
- The earliest release to stop publishing ZetaSQL artifacts is pushed
further, not earlier than 1 full quarter before the next Release (2.65.0 +
3 months would be 2.68.0), and only when it causes issues on maintenance.
Thanks,
Yi
On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
> +1 to this deprecation. Thanks for putting together a clear summary.
>
> FWIW it also has significantly worse performance than Calcite SQL dialect,
> since it calls out to a ZetaSQL subprocess for most calculations, and that
> is less optimized than Beam's Fn API.
>
> Kenn
>
> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
> user@beam.apache.org> wrote:
>
>> I'm in favor of deprecating this and cleaning it up, but it depends on
>> usage. I suspect it is low (or possibly non-existent, especially as there's
>> little upside to moving away from the default). I cc'd user@ just in
>> case anyone wants to chime in there. This may be a good thing to add to our
>> release notes as well (perhaps we can get it in the one that's just about
>> to go out).
>>
>> Unless there is strong, justified pushback, I'd get the deprecation
>> status (e.g. on the javadocs, website) right away. For actual removal, I
>> agree with the idea of waiting until it actually causes issues or we move
>> to the next major beam release, though I might push back at 2.66 being a
>> bit too quick even if the first condition is hit before then and might give
>> people at least a quarter's notice.
>>
>> - Robert
>>
>>
>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>> component [1].
>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>
>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>> early 2022 (See change history [3])
>>>
>>> - Despite incomplete support status, there is no new bug / feature
>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>> minimal adoption [4]
>>>
>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>> with other google dependencies, as a result ZetaSQL component introduces
>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>
>>> - One of the main reason that using ZetaSQL dialect, per [2], was because
>>>
>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
>>> pipelines that write to or read from BigQuery tables.
>>>
>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>
>>> For these reasons, I propose initiating the process of deprecating
>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>
>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>> component in javadoc, beam website, currently I recommend do it in the
>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>> 2025)
>>>
>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>>> and I think we can leave the deprecated status as is until the following
>>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>>> (cut Jun 11, 2025)
>>>
>>> - Continued support for ZetaSQL component involving significant burdens,
>>> like conflict with other Beam dependencies, supported Java versions, etc, or
>>> - When Beam moved to the next release major release (3)
>>>
>>> Thanks for your attention, and any input welcomed!
>>>
>>> Regards,
>>> Yi
>>>
>>> [1]
>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>> [3]
>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>> [4]
>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>> [5] https://github.com/apache/beam/pull/32902
>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Yi Hu via user <us...@beam.apache.org>.
Thanks for inputs! From discussion we have agreed on moving forward
Next step:
- Draft PR to note the deprecation status in Documentation, including
CHANGES, Javadoc, etc
- The earliest release to stop publishing ZetaSQL artifacts is pushed
further, not earlier than 1 full quarter before the next Release (2.65.0 +
3 months would be 2.68.0), and only when it causes issues on maintenance.
Thanks,
Yi
On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <ke...@apache.org> wrote:
> +1 to this deprecation. Thanks for putting together a clear summary.
>
> FWIW it also has significantly worse performance than Calcite SQL dialect,
> since it calls out to a ZetaSQL subprocess for most calculations, and that
> is less optimized than Beam's Fn API.
>
> Kenn
>
> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
> user@beam.apache.org> wrote:
>
>> I'm in favor of deprecating this and cleaning it up, but it depends on
>> usage. I suspect it is low (or possibly non-existent, especially as there's
>> little upside to moving away from the default). I cc'd user@ just in
>> case anyone wants to chime in there. This may be a good thing to add to our
>> release notes as well (perhaps we can get it in the one that's just about
>> to go out).
>>
>> Unless there is strong, justified pushback, I'd get the deprecation
>> status (e.g. on the javadocs, website) right away. For actual removal, I
>> agree with the idea of waiting until it actually causes issues or we move
>> to the next major beam release, though I might push back at 2.66 being a
>> bit too quick even if the first condition is hit before then and might give
>> people at least a quarter's notice.
>>
>> - Robert
>>
>>
>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>>> component [1].
>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>>
>>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>>> early 2022 (See change history [3])
>>>
>>> - Despite incomplete support status, there is no new bug / feature
>>> request opened ever since we migrated to use GitHub Issue, suggesting
>>> minimal adoption [4]
>>>
>>> - We still need to keep zetasql up-to-date if its dependency conflicts
>>> with other google dependencies, as a result ZetaSQL component introduces
>>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>>
>>> - One of the main reason that using ZetaSQL dialect, per [2], was because
>>>
>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
>>> pipelines that write to or read from BigQuery tables.
>>>
>>> As of today, as GCP BigQuery now supports using GoogleSQL
>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery
>>> via BigQuery Connections API / Federated query [6, 7]. This largely
>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery.
>>>
>>> For these reasons, I propose initiating the process of deprecating
>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>>
>>> Firstly, agree on when to document the deprecated status for ZetaSQL
>>> component in javadoc, beam website, currently I recommend do it in the
>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>>> 2025)
>>>
>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>>> and I think we can leave the deprecated status as is until the following
>>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>>> (cut Jun 11, 2025)
>>>
>>> - Continued support for ZetaSQL component involving significant burdens,
>>> like conflict with other Beam dependencies, supported Java versions, etc, or
>>> - When Beam moved to the next release major release (3)
>>>
>>> Thanks for your attention, and any input welcomed!
>>>
>>> Regards,
>>> Yi
>>>
>>> [1]
>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>>> [3]
>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>>> [4]
>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>>> [5] https://github.com/apache/beam/pull/32902
>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Kenneth Knowles <ke...@apache.org>.
+1 to this deprecation. Thanks for putting together a clear summary.
FWIW it also has significantly worse performance than Calcite SQL dialect,
since it calls out to a ZetaSQL subprocess for most calculations, and that
is less optimized than Beam's Fn API.
Kenn
On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
user@beam.apache.org> wrote:
> I'm in favor of deprecating this and cleaning it up, but it depends on
> usage. I suspect it is low (or possibly non-existent, especially as there's
> little upside to moving away from the default). I cc'd user@ just in case
> anyone wants to chime in there. This may be a good thing to add to our
> release notes as well (perhaps we can get it in the one that's just about
> to go out).
>
> Unless there is strong, justified pushback, I'd get the deprecation status
> (e.g. on the javadocs, website) right away. For actual removal, I agree
> with the idea of waiting until it actually causes issues or we move to the
> next major beam release, though I might push back at 2.66 being a bit too
> quick even if the first condition is hit before then and might give people
> at least a quarter's notice.
>
> - Robert
>
>
> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org> wrote:
>
>> Hi everyone,
>>
>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>> component [1].
>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>
>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>> early 2022 (See change history [3])
>>
>> - Despite incomplete support status, there is no new bug / feature
>> request opened ever since we migrated to use GitHub Issue, suggesting
>> minimal adoption [4]
>>
>> - We still need to keep zetasql up-to-date if its dependency conflicts
>> with other google dependencies, as a result ZetaSQL component introduces
>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>
>> - One of the main reason that using ZetaSQL dialect, per [2], was because
>>
>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
>> pipelines that write to or read from BigQuery tables.
>>
>> As of today, as GCP BigQuery now supports using GoogleSQL (open-sourced
>> as ZetaSQL) querying data that's stored outside of BigQuery via BigQuery
>> Connections API / Federated query [6, 7]. This largely provides an
>> alternative for using Beam's ZetaSQL interacting with BigQuery.
>>
>> For these reasons, I propose initiating the process of deprecating
>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>
>> Firstly, agree on when to document the deprecated status for ZetaSQL
>> component in javadoc, beam website, currently I recommend do it in the
>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>> 2025)
>>
>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>> and I think we can leave the deprecated status as is until the following
>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>> (cut Jun 11, 2025)
>>
>> - Continued support for ZetaSQL component involving significant burdens,
>> like conflict with other Beam dependencies, supported Java versions, etc, or
>> - When Beam moved to the next release major release (3)
>>
>> Thanks for your attention, and any input welcomed!
>>
>> Regards,
>> Yi
>>
>> [1]
>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>> [3]
>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>> [4]
>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>> [5] https://github.com/apache/beam/pull/32902
>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Kenneth Knowles <ke...@apache.org>.
+1 to this deprecation. Thanks for putting together a clear summary.
FWIW it also has significantly worse performance than Calcite SQL dialect,
since it calls out to a ZetaSQL subprocess for most calculations, and that
is less optimized than Beam's Fn API.
Kenn
On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user <
user@beam.apache.org> wrote:
> I'm in favor of deprecating this and cleaning it up, but it depends on
> usage. I suspect it is low (or possibly non-existent, especially as there's
> little upside to moving away from the default). I cc'd user@ just in case
> anyone wants to chime in there. This may be a good thing to add to our
> release notes as well (perhaps we can get it in the one that's just about
> to go out).
>
> Unless there is strong, justified pushback, I'd get the deprecation status
> (e.g. on the javadocs, website) right away. For actual removal, I agree
> with the idea of waiting until it actually causes issues or we move to the
> next major beam release, though I might push back at 2.66 being a bit too
> quick even if the first condition is hit before then and might give people
> at least a quarter's notice.
>
> - Robert
>
>
> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org> wrote:
>
>> Hi everyone,
>>
>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
>> component [1].
>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>>
>> - Developments in Beam for ZetaSQL dialect effectively stalled since
>> early 2022 (See change history [3])
>>
>> - Despite incomplete support status, there is no new bug / feature
>> request opened ever since we migrated to use GitHub Issue, suggesting
>> minimal adoption [4]
>>
>> - We still need to keep zetasql up-to-date if its dependency conflicts
>> with other google dependencies, as a result ZetaSQL component introduces
>> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>>
>> - One of the main reason that using ZetaSQL dialect, per [2], was because
>>
>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
>> pipelines that write to or read from BigQuery tables.
>>
>> As of today, as GCP BigQuery now supports using GoogleSQL (open-sourced
>> as ZetaSQL) querying data that's stored outside of BigQuery via BigQuery
>> Connections API / Federated query [6, 7]. This largely provides an
>> alternative for using Beam's ZetaSQL interacting with BigQuery.
>>
>> For these reasons, I propose initiating the process of deprecating
>> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>>
>> Firstly, agree on when to document the deprecated status for ZetaSQL
>> component in javadoc, beam website, currently I recommend do it in the
>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
>> 2025)
>>
>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
>> and I think we can leave the deprecated status as is until the following
>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
>> (cut Jun 11, 2025)
>>
>> - Continued support for ZetaSQL component involving significant burdens,
>> like conflict with other Beam dependencies, supported Java versions, etc, or
>> - When Beam moved to the next release major release (3)
>>
>> Thanks for your attention, and any input welcomed!
>>
>> Regards,
>> Yi
>>
>> [1]
>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
>> [2] https://beam.apache.org/documentation/dsls/sql/overview/
>> [3]
>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
>> [4]
>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
>> [5] https://github.com/apache/beam/pull/32902
>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>
Re: [Discussion] Deprecate ZetaSQL
Posted by Robert Bradshaw via user <us...@beam.apache.org>.
I'm in favor of deprecating this and cleaning it up, but it depends on
usage. I suspect it is low (or possibly non-existent, especially as there's
little upside to moving away from the default). I cc'd user@ just in case
anyone wants to chime in there. This may be a good thing to add to our
release notes as well (perhaps we can get it in the one that's just about
to go out).
Unless there is strong, justified pushback, I'd get the deprecation status
(e.g. on the javadocs, website) right away. For actual removal, I agree
with the idea of waiting until it actually causes issues or we move to the
next major beam release, though I might push back at 2.66 being a bit too
quick even if the first condition is hit before then and might give people
at least a quarter's notice.
- Robert
On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org> wrote:
> Hi everyone,
>
> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
> component [1].
> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>
> - Developments in Beam for ZetaSQL dialect effectively stalled since early
> 2022 (See change history [3])
>
> - Despite incomplete support status, there is no new bug / feature request
> opened ever since we migrated to use GitHub Issue, suggesting minimal
> adoption [4]
>
> - We still need to keep zetasql up-to-date if its dependency conflicts
> with other google dependencies, as a result ZetaSQL component introduces
> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>
> - One of the main reason that using ZetaSQL dialect, per [2], was because
>
> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
> pipelines that write to or read from BigQuery tables.
>
> As of today, as GCP BigQuery now supports using GoogleSQL (open-sourced
> as ZetaSQL) querying data that's stored outside of BigQuery via BigQuery
> Connections API / Federated query [6, 7]. This largely provides an
> alternative for using Beam's ZetaSQL interacting with BigQuery.
>
> For these reasons, I propose initiating the process of deprecating
> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>
> Firstly, agree on when to document the deprecated status for ZetaSQL
> component in javadoc, beam website, currently I recommend do it in the
> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
> 2025)
>
> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
> and I think we can leave the deprecated status as is until the following
> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
> (cut Jun 11, 2025)
>
> - Continued support for ZetaSQL component involving significant burdens,
> like conflict with other Beam dependencies, supported Java versions, etc, or
> - When Beam moved to the next release major release (3)
>
> Thanks for your attention, and any input welcomed!
>
> Regards,
> Yi
>
> [1]
> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
> [2] https://beam.apache.org/documentation/dsls/sql/overview/
> [3]
> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
> [4]
> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
> [5] https://github.com/apache/beam/pull/32902
> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>
Re: [Discussion] Deprecate ZetaSQL
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
I'm in favor of deprecating this and cleaning it up, but it depends on
usage. I suspect it is low (or possibly non-existent, especially as there's
little upside to moving away from the default). I cc'd user@ just in case
anyone wants to chime in there. This may be a good thing to add to our
release notes as well (perhaps we can get it in the one that's just about
to go out).
Unless there is strong, justified pushback, I'd get the deprecation status
(e.g. on the javadocs, website) right away. For actual removal, I agree
with the idea of waiting until it actually causes issues or we move to the
next major beam release, though I might push back at 2.66 being a bit too
quick even if the first condition is hit before then and might give people
at least a quarter's notice.
- Robert
On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <de...@beam.apache.org> wrote:
> Hi everyone,
>
> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL
> component [1].
> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and
> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons
>
> - Developments in Beam for ZetaSQL dialect effectively stalled since early
> 2022 (See change history [3])
>
> - Despite incomplete support status, there is no new bug / feature request
> opened ever since we migrated to use GitHub Issue, suggesting minimal
> adoption [4]
>
> - We still need to keep zetasql up-to-date if its dependency conflicts
> with other google dependencies, as a result ZetaSQL component introduces
> maintenance burden when upgrading GCP-BOM (e.g. [5]).
>
> - One of the main reason that using ZetaSQL dialect, per [2], was because
>
> > ZetaSQL is more compatible with BigQuery, so it’s especially useful in
> pipelines that write to or read from BigQuery tables.
>
> As of today, as GCP BigQuery now supports using GoogleSQL (open-sourced
> as ZetaSQL) querying data that's stored outside of BigQuery via BigQuery
> Connections API / Federated query [6, 7]. This largely provides an
> alternative for using Beam's ZetaSQL interacting with BigQuery.
>
> For these reasons, I propose initiating the process of deprecating
> Beam SQL's ZetaSQL component. There are two decisions needed to be made:
>
> Firstly, agree on when to document the deprecated status for ZetaSQL
> component in javadoc, beam website, currently I recommend do it in the
> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30,
> 2025)
>
> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change,
> and I think we can leave the deprecated status as is until the following
> situation emerges, whichever comes first, and no earlier than Beam 2.66.0
> (cut Jun 11, 2025)
>
> - Continued support for ZetaSQL component involving significant burdens,
> like conflict with other Beam dependencies, supported Java versions, etc, or
> - When Beam moved to the next release major release (3)
>
> Thanks for your attention, and any input welcomed!
>
> Regards,
> Yi
>
> [1]
> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql
> [2] https://beam.apache.org/documentation/dsls/sql/overview/
> [3]
> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
> [4]
> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20
> [5] https://github.com/apache/beam/pull/32902
> [6] https://cloud.google.com/bigquery/docs/connections-api-intro
> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>