You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ahmed Abualsaud via dev <de...@beam.apache.org> on 2026/10/23 03:32:22 UTC

[PROPOSAL] Add PRIMARY KEY to Beam SQL

Hey everyone,

Following up on the last proposal, this one allows users to create a table
with a Primary Key or modify an existing table's Primary Key using Beam SQL.

https://docs.google.com/document/d/1lGynrfHlyvcmmyzJZ2gkL9zK8KF1ZnfkcImnMmM_ciw/edit

This is a necessary piece to facilitate using CDC in Beam SQL. Looking for
feedback and suggestions!

Best,
Ahmed

Re: [PROPOSAL] Add PRIMARY KEY to Beam SQL

Posted by Ahmed Abualsaud via dev <de...@beam.apache.org>.
Yeah it's true most of the stuff we're adding is standard SQL syntax. Most
of the real work is just mapping it to a specific table format and the
relevant connectors.

> Is there a way we can do this "all at once" ... or TBH integrating with
something

Well we're already integrated with Apache Calcite (similar to Flink), which
gives us a lot of free parsing logic and object representation. Our
Catalog/Database/Table representations all implement Calcite's Schema
interface so they make use of Calcite's parsing, validation, planning.

Beyond that, AFAIK every engine defines its own interface for applying DDL
to a data source. For example Flink's Catalog API
<https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/table/catalog/Catalog.html>
and
Spark's TableCatalog API
<https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/connector/catalog/TableCatalog.html>.
From what I know, there isn't a shared DDL mapping layer that we can plug
into.

> we shouldn't need many designs or discussion around it

I think the underlying integration work is implementation detail and in
that sense agree it shouldn't require a discussion. But I think syntax
changes deserve at least a callout since SQL is such a high-level,
user-facing language. There are also many devs here with stronger SQL
experience and more insightful opinions than mine (for example, last
thread's discussion turned the CHANGES clause into a function instead).
Although maybe this time PRIMARY KEY is a no-brainer.

Generally I'm trying to keep these proposals short and focused only on the
syntax additions, with some comparison to other dialects.


Best,
Ahmed


On Fri, Oct 23, 2026 at 10:48 AM Kenneth Knowles <ke...@apache.org> wrote:

> Seems clearly useful.
>
> Just to derail the thread... :-) what is the eventual endpoint of the Beam
> SQL DDL work like this? It seems like we are probably duplicating something
> that has been done many times before. Namely: having a model of what a
> table is, and a way to map DDL to specific implementations that have
> different backing connectors.
>
> I think it still makes sense to have our SQL CLI / driver or other things
> built on Beam SQL be able to "do it all" because it is a whole-product
> entrypoint. And of course many many sibling projects have their own full
> implementation of all this stuff.
>
> Is there a way we can do this "all at once" and be done with it, having
> the whole "you type SQL into Beam and we know how to do the thing to the
> underlying storage / catalog" step? It feels like we shouldn't need many
> designs or discussion around it, unless it has Beam-specific
> considerations, and we could possibly crank it out semi-automatically? Or
> TBH integrating with something is better than vibe coding, so as to get
> maintenance and improvements for free.
>
> Kenn
>
> On Thu, Oct 22, 2026 at 11:33 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Hey everyone,
>>
>> Following up on the last proposal, this one allows users to create a
>> table with a Primary Key or modify an existing table's Primary Key using
>> Beam SQL.
>>
>>
>> https://docs.google.com/document/d/1lGynrfHlyvcmmyzJZ2gkL9zK8KF1ZnfkcImnMmM_ciw/edit
>>
>> This is a necessary piece to facilitate using CDC in Beam SQL. Looking
>> for feedback and suggestions!
>>
>> Best,
>> Ahmed
>>
>

Re: [PROPOSAL] Add PRIMARY KEY to Beam SQL

Posted by Kenneth Knowles <ke...@apache.org>.
Seems clearly useful.

Just to derail the thread... :-) what is the eventual endpoint of the Beam
SQL DDL work like this? It seems like we are probably duplicating something
that has been done many times before. Namely: having a model of what a
table is, and a way to map DDL to specific implementations that have
different backing connectors.

I think it still makes sense to have our SQL CLI / driver or other things
built on Beam SQL be able to "do it all" because it is a whole-product
entrypoint. And of course many many sibling projects have their own full
implementation of all this stuff.

Is there a way we can do this "all at once" and be done with it, having the
whole "you type SQL into Beam and we know how to do the thing to the
underlying storage / catalog" step? It feels like we shouldn't need many
designs or discussion around it, unless it has Beam-specific
considerations, and we could possibly crank it out semi-automatically? Or
TBH integrating with something is better than vibe coding, so as to get
maintenance and improvements for free.

Kenn

On Thu, Oct 22, 2026 at 11:33 PM Ahmed Abualsaud via dev <
dev@beam.apache.org> wrote:

> Hey everyone,
>
> Following up on the last proposal, this one allows users to create a table
> with a Primary Key or modify an existing table's Primary Key using Beam SQL.
>
>
> https://docs.google.com/document/d/1lGynrfHlyvcmmyzJZ2gkL9zK8KF1ZnfkcImnMmM_ciw/edit
>
> This is a necessary piece to facilitate using CDC in Beam SQL. Looking
> for feedback and suggestions!
>
> Best,
> Ahmed
>