[DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Ferenc Csaky via dev <de...@flink.apache.org>]

Hello devs,

I would like to discuss a question around TO_CHANGELOG. The PTF turns a dynamic
table into an insert-only table and prepends an explicit operation-code column.
PR #29102 proposes an optional argument to omit that column and preserve the
input schema [1]. It would default to the current behavior, so existing queries
and compiled plans remain compatible.

The motivating use case is a generalized Flink SQL job used with tables of
different schemas. It transforms a retract stream into a special upsert stream
that retains only +I and +U; deletion information is already represented in
the data schema for historical reasons. Since the schemas are not known in
advance, `SELECT *` is the practical projection. Retaining the schema avoids a
custom UDF solely for removing the extra operation column.

There are two possible designs:

1. Add an optional TO_CHANGELOG parameter, for example
   `include_op_column => FALSE`, to omit the operation column.
2. Keep the PTF API as it is and use the SQL projection that will become
   available with the Calcite 1.42 upgrade:

   SELECT * EXCLUDE(op)
   FROM TO_CHANGELOG(input => TABLE source_table, ...);

Omitting the operation column does not make a retract stream append-only in
the usual sense. Callers that receive updates or deletes need an appropriate
representation or filtering strategy; otherwise different change events can
look identical after the operation code is removed. The motivating pipeline
handles this separately.

While `SELECT * EXCLUDE(op)` would solve this specific use case, I would prefer
changing the PTF. Requiring TO_CHANGELOG to change the output schema feels like
a sometimes unnecessary restriction. An option with a reasonable default adds
little complexity, and most users do not even need to know about it, while it
leaves more room for flexibility to retain the input schema with `SELECT *`
when needed.

WDYT?

Best,
Ferenc

[1] https://github.com/apache/flink/pull/29102

Re: [DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Timo Walther <tw...@apache.org>]

Hi everyone,

thanks for starting the discussion Gustavo. On one side, I understand 
the need for having this convenience feature, but on the other side I 
see the following downsides of it:

1. The function is called TO_CHANGELOG, a changelog needs an op field. 
Omitting it leaves a false impression that a changelog suddenly becomes 
an insert-only pipeline.
2. For all functions, we should avoid overloading functions too heavily. 
The number and shape of parameters should be well-defined to avoid 
confusion and reduce documentation and special cases in implementation. 
Keep the feature matrix dense.
3. A PTF should not handle cases that can be easily done in SQL.

So overall I agree with Gustavo, downstream SQL can do the projection. 
In the future, many users might also want to remove the partition key or 
rowtime column that a PTF adds. We should rather put effort in support 
SELECT * EXCLUDE and have better projection means going forward.

Cheers,
Timo



On 21.10.26 12:05, Gustavo de Morais wrote:
> Hi Mate,
> 
> Thanks for contributing to the discussion.
> 
> I agree with 1 and 3. Regarding 2, the Table API already has an equivalent:
> table.toChangelog(...).dropColumns($("op")), or withoutColumns($("op")) in
> a select.
> 
> Kind regards,
> Gustavo
> 
> On Thu, 15 Oct 2026 at 16:48, Mate Czagany <cz...@gmail.com> wrote:
> 
>> Hi everyone,
>>
>> Thank you for initiating this discussion. I support adding the optional
>> argument here.
>>
>> 1. TO_CHANGELOG already decides what appears in the output with op_mapping
>> and produces_full_deletes. Whether the code column is emitted at all is the
>> same kind of decision
>>
>> 2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom
>> between SQL and Table API.
>>
>> 3. The cost is minimal. The PR is additive, defaults to the current
>> behavior, and most users will never see the argument.
>>
>> Overall, +1 as long as we correctly document in the include_op_column
>> section that -U/-D rows will look like inserts and will show up as
>> duplicate records, and recommend pairing it with an op_mapping that only
>> keeps INSERT and UPDATE_AFTER.
>>
>> Best regards,
>> Mate
>>
>> On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <gu...@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Bumping this one. I'd be happy if we have other opinions here.
>>>
>>> Kind regards,
>>> Gustavo
>>>
>>> On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <gu...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ferenc,
>>>>
>>>> Thanks for the thread. I'm genuinely interested in the community's
>>> opinion
>>>> on an additional include_op_column. I see three scenarios:
>>>>
>>>> 1. The one you mentioned:
>>>> A retract table produces +I, +U, -U, -D events and users want to
>> convert
>>>> it to append only. After turning it into an append stream, the user
>> might
>>>> be interested only in +I and +U and will use the added "op" column to
>>>> filter out -U and -D. This generally requires the "op" column. In your
>>>> case, the deletion information is already represented in the data
>> schema
>>>> for historical reasons, so you don't need the "op" column. I wonder how
>>>> common that is and if we should add a new parameter for this specific
>>> case,
>>>> or if we are fine with the EXCLUDE idiomatic approach, since this
>> doesn't
>>>> sound like a common case.
>>>>
>>>> 2. The user is certain that their updating table will never contain
>>> either
>>>> -U or -D due to the nature of the data. They want to keep the schema,
>> and
>>>> the op column is not necessary for their use case.
>>>>
>>>> 3.A retract table produces +I, +U, -U, -D events and the user only
>> needs
>>>> +I and +U but doesn't mind the duplicated records that will appear due
>> to
>>>> -D and -U. This is less efficient and will create unnecessary duplicate
>>>> records. However, the user doesn't mind.
>>>>
>>>> Looking forward to hearing what the community thinks is the most common
>>>> case here, and whether it's worth a dedicated parameter.
>>>>
>>>> Kind regards,
>>>> Gustavo
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <
>> dev@flink.apache.org>
>>>> wrote:
>>>>
>>>>> Hello devs,
>>>>>
>>>>> I would like to discuss a question around TO_CHANGELOG. The PTF turns
>> a
>>>>> dynamic
>>>>> table into an insert-only table and prepends an explicit
>> operation-code
>>>>> column.
>>>>> PR #29102 proposes an optional argument to omit that column and
>> preserve
>>>>> the
>>>>> input schema [1]. It would default to the current behavior, so
>> existing
>>>>> queries
>>>>> and compiled plans remain compatible.
>>>>>
>>>>> The motivating use case is a generalized Flink SQL job used with
>> tables
>>> of
>>>>> different schemas. It transforms a retract stream into a special
>> upsert
>>>>> stream
>>>>> that retains only +I and +U; deletion information is already
>> represented
>>>>> in
>>>>> the data schema for historical reasons. Since the schemas are not
>> known
>>> in
>>>>> advance, `SELECT *` is the practical projection. Retaining the schema
>>>>> avoids a
>>>>> custom UDF solely for removing the extra operation column.
>>>>>
>>>>> There are two possible designs:
>>>>>
>>>>> 1. Add an optional TO_CHANGELOG parameter, for example
>>>>>     `include_op_column => FALSE`, to omit the operation column.
>>>>> 2. Keep the PTF API as it is and use the SQL projection that will
>> become
>>>>>     available with the Calcite 1.42 upgrade:
>>>>>
>>>>>     SELECT * EXCLUDE(op)
>>>>>     FROM TO_CHANGELOG(input => TABLE source_table, ...);
>>>>>
>>>>> Omitting the operation column does not make a retract stream
>> append-only
>>>>> in
>>>>> the usual sense. Callers that receive updates or deletes need an
>>>>> appropriate
>>>>> representation or filtering strategy; otherwise different change
>> events
>>>>> can
>>>>> look identical after the operation code is removed. The motivating
>>>>> pipeline
>>>>> handles this separately.
>>>>>
>>>>> While `SELECT * EXCLUDE(op)` would solve this specific use case, I
>> would
>>>>> prefer
>>>>> changing the PTF. Requiring TO_CHANGELOG to change the output schema
>>>>> feels like
>>>>> a sometimes unnecessary restriction. An option with a reasonable
>> default
>>>>> adds
>>>>> little complexity, and most users do not even need to know about it,
>>>>> while it
>>>>> leaves more room for flexibility to retain the input schema with
>> `SELECT
>>>>> *`
>>>>> when needed.
>>>>>
>>>>> WDYT?
>>>>>
>>>>> Best,
>>>>> Ferenc
>>>>>
>>>>> [1] https://github.com/apache/flink/pull/29102
>>>>>
>>>>
>>>
>>
> 

Re: [DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Gustavo de Morais <gu...@gmail.com>]

Hi Mate,

Thanks for contributing to the discussion.

I agree with 1 and 3. Regarding 2, the Table API already has an equivalent:
table.toChangelog(...).dropColumns($("op")), or withoutColumns($("op")) in
a select.

Kind regards,
Gustavo

On Thu, 15 Oct 2026 at 16:48, Mate Czagany <cz...@gmail.com> wrote:

> Hi everyone,
>
> Thank you for initiating this discussion. I support adding the optional
> argument here.
>
> 1. TO_CHANGELOG already decides what appears in the output with op_mapping
> and produces_full_deletes. Whether the code column is emitted at all is the
> same kind of decision
>
> 2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom
> between SQL and Table API.
>
> 3. The cost is minimal. The PR is additive, defaults to the current
> behavior, and most users will never see the argument.
>
> Overall, +1 as long as we correctly document in the include_op_column
> section that -U/-D rows will look like inserts and will show up as
> duplicate records, and recommend pairing it with an op_mapping that only
> keeps INSERT and UPDATE_AFTER.
>
> Best regards,
> Mate
>
> On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <gu...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Bumping this one. I'd be happy if we have other opinions here.
> >
> > Kind regards,
> > Gustavo
> >
> > On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <gu...@gmail.com>
> > wrote:
> >
> > > Hi Ferenc,
> > >
> > > Thanks for the thread. I'm genuinely interested in the community's
> > opinion
> > > on an additional include_op_column. I see three scenarios:
> > >
> > > 1. The one you mentioned:
> > > A retract table produces +I, +U, -U, -D events and users want to
> convert
> > > it to append only. After turning it into an append stream, the user
> might
> > > be interested only in +I and +U and will use the added "op" column to
> > > filter out -U and -D. This generally requires the "op" column. In your
> > > case, the deletion information is already represented in the data
> schema
> > > for historical reasons, so you don't need the "op" column. I wonder how
> > > common that is and if we should add a new parameter for this specific
> > case,
> > > or if we are fine with the EXCLUDE idiomatic approach, since this
> doesn't
> > > sound like a common case.
> > >
> > > 2. The user is certain that their updating table will never contain
> > either
> > > -U or -D due to the nature of the data. They want to keep the schema,
> and
> > > the op column is not necessary for their use case.
> > >
> > > 3.A retract table produces +I, +U, -U, -D events and the user only
> needs
> > > +I and +U but doesn't mind the duplicated records that will appear due
> to
> > > -D and -U. This is less efficient and will create unnecessary duplicate
> > > records. However, the user doesn't mind.
> > >
> > > Looking forward to hearing what the community thinks is the most common
> > > case here, and whether it's worth a dedicated parameter.
> > >
> > > Kind regards,
> > > Gustavo
> > >
> > >
> > >
> > >
> > > On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <
> dev@flink.apache.org>
> > > wrote:
> > >
> > >> Hello devs,
> > >>
> > >> I would like to discuss a question around TO_CHANGELOG. The PTF turns
> a
> > >> dynamic
> > >> table into an insert-only table and prepends an explicit
> operation-code
> > >> column.
> > >> PR #29102 proposes an optional argument to omit that column and
> preserve
> > >> the
> > >> input schema [1]. It would default to the current behavior, so
> existing
> > >> queries
> > >> and compiled plans remain compatible.
> > >>
> > >> The motivating use case is a generalized Flink SQL job used with
> tables
> > of
> > >> different schemas. It transforms a retract stream into a special
> upsert
> > >> stream
> > >> that retains only +I and +U; deletion information is already
> represented
> > >> in
> > >> the data schema for historical reasons. Since the schemas are not
> known
> > in
> > >> advance, `SELECT *` is the practical projection. Retaining the schema
> > >> avoids a
> > >> custom UDF solely for removing the extra operation column.
> > >>
> > >> There are two possible designs:
> > >>
> > >> 1. Add an optional TO_CHANGELOG parameter, for example
> > >>    `include_op_column => FALSE`, to omit the operation column.
> > >> 2. Keep the PTF API as it is and use the SQL projection that will
> become
> > >>    available with the Calcite 1.42 upgrade:
> > >>
> > >>    SELECT * EXCLUDE(op)
> > >>    FROM TO_CHANGELOG(input => TABLE source_table, ...);
> > >>
> > >> Omitting the operation column does not make a retract stream
> append-only
> > >> in
> > >> the usual sense. Callers that receive updates or deletes need an
> > >> appropriate
> > >> representation or filtering strategy; otherwise different change
> events
> > >> can
> > >> look identical after the operation code is removed. The motivating
> > >> pipeline
> > >> handles this separately.
> > >>
> > >> While `SELECT * EXCLUDE(op)` would solve this specific use case, I
> would
> > >> prefer
> > >> changing the PTF. Requiring TO_CHANGELOG to change the output schema
> > >> feels like
> > >> a sometimes unnecessary restriction. An option with a reasonable
> default
> > >> adds
> > >> little complexity, and most users do not even need to know about it,
> > >> while it
> > >> leaves more room for flexibility to retain the input schema with
> `SELECT
> > >> *`
> > >> when needed.
> > >>
> > >> WDYT?
> > >>
> > >> Best,
> > >> Ferenc
> > >>
> > >> [1] https://github.com/apache/flink/pull/29102
> > >>
> > >
> >
>

Re: [DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Mate Czagany <cz...@gmail.com>]

Hi everyone,

Thank you for initiating this discussion. I support adding the optional
argument here.

1. TO_CHANGELOG already decides what appears in the output with op_mapping
and produces_full_deletes. Whether the code column is emitted at all is the
same kind of decision

2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom
between SQL and Table API.

3. The cost is minimal. The PR is additive, defaults to the current
behavior, and most users will never see the argument.

Overall, +1 as long as we correctly document in the include_op_column
section that -U/-D rows will look like inserts and will show up as
duplicate records, and recommend pairing it with an op_mapping that only
keeps INSERT and UPDATE_AFTER.

Best regards,
Mate

On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <gu...@gmail.com>
wrote:

> Hi everyone,
>
> Bumping this one. I'd be happy if we have other opinions here.
>
> Kind regards,
> Gustavo
>
> On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <gu...@gmail.com>
> wrote:
>
> > Hi Ferenc,
> >
> > Thanks for the thread. I'm genuinely interested in the community's
> opinion
> > on an additional include_op_column. I see three scenarios:
> >
> > 1. The one you mentioned:
> > A retract table produces +I, +U, -U, -D events and users want to convert
> > it to append only. After turning it into an append stream, the user might
> > be interested only in +I and +U and will use the added "op" column to
> > filter out -U and -D. This generally requires the "op" column. In your
> > case, the deletion information is already represented in the data schema
> > for historical reasons, so you don't need the "op" column. I wonder how
> > common that is and if we should add a new parameter for this specific
> case,
> > or if we are fine with the EXCLUDE idiomatic approach, since this doesn't
> > sound like a common case.
> >
> > 2. The user is certain that their updating table will never contain
> either
> > -U or -D due to the nature of the data. They want to keep the schema, and
> > the op column is not necessary for their use case.
> >
> > 3.A retract table produces +I, +U, -U, -D events and the user only needs
> > +I and +U but doesn't mind the duplicated records that will appear due to
> > -D and -U. This is less efficient and will create unnecessary duplicate
> > records. However, the user doesn't mind.
> >
> > Looking forward to hearing what the community thinks is the most common
> > case here, and whether it's worth a dedicated parameter.
> >
> > Kind regards,
> > Gustavo
> >
> >
> >
> >
> > On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <de...@flink.apache.org>
> > wrote:
> >
> >> Hello devs,
> >>
> >> I would like to discuss a question around TO_CHANGELOG. The PTF turns a
> >> dynamic
> >> table into an insert-only table and prepends an explicit operation-code
> >> column.
> >> PR #29102 proposes an optional argument to omit that column and preserve
> >> the
> >> input schema [1]. It would default to the current behavior, so existing
> >> queries
> >> and compiled plans remain compatible.
> >>
> >> The motivating use case is a generalized Flink SQL job used with tables
> of
> >> different schemas. It transforms a retract stream into a special upsert
> >> stream
> >> that retains only +I and +U; deletion information is already represented
> >> in
> >> the data schema for historical reasons. Since the schemas are not known
> in
> >> advance, `SELECT *` is the practical projection. Retaining the schema
> >> avoids a
> >> custom UDF solely for removing the extra operation column.
> >>
> >> There are two possible designs:
> >>
> >> 1. Add an optional TO_CHANGELOG parameter, for example
> >>    `include_op_column => FALSE`, to omit the operation column.
> >> 2. Keep the PTF API as it is and use the SQL projection that will become
> >>    available with the Calcite 1.42 upgrade:
> >>
> >>    SELECT * EXCLUDE(op)
> >>    FROM TO_CHANGELOG(input => TABLE source_table, ...);
> >>
> >> Omitting the operation column does not make a retract stream append-only
> >> in
> >> the usual sense. Callers that receive updates or deletes need an
> >> appropriate
> >> representation or filtering strategy; otherwise different change events
> >> can
> >> look identical after the operation code is removed. The motivating
> >> pipeline
> >> handles this separately.
> >>
> >> While `SELECT * EXCLUDE(op)` would solve this specific use case, I would
> >> prefer
> >> changing the PTF. Requiring TO_CHANGELOG to change the output schema
> >> feels like
> >> a sometimes unnecessary restriction. An option with a reasonable default
> >> adds
> >> little complexity, and most users do not even need to know about it,
> >> while it
> >> leaves more room for flexibility to retain the input schema with `SELECT
> >> *`
> >> when needed.
> >>
> >> WDYT?
> >>
> >> Best,
> >> Ferenc
> >>
> >> [1] https://github.com/apache/flink/pull/29102
> >>
> >
>

Re: [DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Gustavo de Morais <gu...@gmail.com>]

Hi everyone,

Bumping this one. I'd be happy if we have other opinions here.

Kind regards,
Gustavo

On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <gu...@gmail.com>
wrote:

> Hi Ferenc,
>
> Thanks for the thread. I'm genuinely interested in the community's opinion
> on an additional include_op_column. I see three scenarios:
>
> 1. The one you mentioned:
> A retract table produces +I, +U, -U, -D events and users want to convert
> it to append only. After turning it into an append stream, the user might
> be interested only in +I and +U and will use the added "op" column to
> filter out -U and -D. This generally requires the "op" column. In your
> case, the deletion information is already represented in the data schema
> for historical reasons, so you don't need the "op" column. I wonder how
> common that is and if we should add a new parameter for this specific case,
> or if we are fine with the EXCLUDE idiomatic approach, since this doesn't
> sound like a common case.
>
> 2. The user is certain that their updating table will never contain either
> -U or -D due to the nature of the data. They want to keep the schema, and
> the op column is not necessary for their use case.
>
> 3.A retract table produces +I, +U, -U, -D events and the user only needs
> +I and +U but doesn't mind the duplicated records that will appear due to
> -D and -U. This is less efficient and will create unnecessary duplicate
> records. However, the user doesn't mind.
>
> Looking forward to hearing what the community thinks is the most common
> case here, and whether it's worth a dedicated parameter.
>
> Kind regards,
> Gustavo
>
>
>
>
> On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <de...@flink.apache.org>
> wrote:
>
>> Hello devs,
>>
>> I would like to discuss a question around TO_CHANGELOG. The PTF turns a
>> dynamic
>> table into an insert-only table and prepends an explicit operation-code
>> column.
>> PR #29102 proposes an optional argument to omit that column and preserve
>> the
>> input schema [1]. It would default to the current behavior, so existing
>> queries
>> and compiled plans remain compatible.
>>
>> The motivating use case is a generalized Flink SQL job used with tables of
>> different schemas. It transforms a retract stream into a special upsert
>> stream
>> that retains only +I and +U; deletion information is already represented
>> in
>> the data schema for historical reasons. Since the schemas are not known in
>> advance, `SELECT *` is the practical projection. Retaining the schema
>> avoids a
>> custom UDF solely for removing the extra operation column.
>>
>> There are two possible designs:
>>
>> 1. Add an optional TO_CHANGELOG parameter, for example
>>    `include_op_column => FALSE`, to omit the operation column.
>> 2. Keep the PTF API as it is and use the SQL projection that will become
>>    available with the Calcite 1.42 upgrade:
>>
>>    SELECT * EXCLUDE(op)
>>    FROM TO_CHANGELOG(input => TABLE source_table, ...);
>>
>> Omitting the operation column does not make a retract stream append-only
>> in
>> the usual sense. Callers that receive updates or deletes need an
>> appropriate
>> representation or filtering strategy; otherwise different change events
>> can
>> look identical after the operation code is removed. The motivating
>> pipeline
>> handles this separately.
>>
>> While `SELECT * EXCLUDE(op)` would solve this specific use case, I would
>> prefer
>> changing the PTF. Requiring TO_CHANGELOG to change the output schema
>> feels like
>> a sometimes unnecessary restriction. An option with a reasonable default
>> adds
>> little complexity, and most users do not even need to know about it,
>> while it
>> leaves more room for flexibility to retain the input schema with `SELECT
>> *`
>> when needed.
>>
>> WDYT?
>>
>> Best,
>> Ferenc
>>
>> [1] https://github.com/apache/flink/pull/29102
>>
>

Re: [DISCUSS] TO_CHANGELOG: Add optional operation column or EXCLUDE projection?

[Gustavo de Morais <gu...@gmail.com>]

Hi Ferenc,

Thanks for the thread. I'm genuinely interested in the community's opinion
on an additional include_op_column. I see three scenarios:

1. The one you mentioned:
A retract table produces +I, +U, -U, -D events and users want to convert it
to append only. After turning it into an append stream, the user might be
interested only in +I and +U and will use the added "op" column to filter
out -U and -D. This generally requires the "op" column. In your case, the
deletion information is already represented in the data schema for
historical reasons, so you don't need the "op" column. I wonder how common
that is and if we should add a new parameter for this specific case, or if
we are fine with the EXCLUDE idiomatic approach, since this doesn't sound
like a common case.

2. The user is certain that their updating table will never contain either
-U or -D due to the nature of the data. They want to keep the schema, and
the op column is not necessary for their use case.

3.A retract table produces +I, +U, -U, -D events and the user only needs +I
and +U but doesn't mind the duplicated records that will appear due to -D
and -U. This is less efficient and will create unnecessary duplicate
records. However, the user doesn't mind.

Looking forward to hearing what the community thinks is the most common
case here, and whether it's worth a dedicated parameter.

Kind regards,
Gustavo




On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <de...@flink.apache.org>
wrote:

> Hello devs,
>
> I would like to discuss a question around TO_CHANGELOG. The PTF turns a
> dynamic
> table into an insert-only table and prepends an explicit operation-code
> column.
> PR #29102 proposes an optional argument to omit that column and preserve
> the
> input schema [1]. It would default to the current behavior, so existing
> queries
> and compiled plans remain compatible.
>
> The motivating use case is a generalized Flink SQL job used with tables of
> different schemas. It transforms a retract stream into a special upsert
> stream
> that retains only +I and +U; deletion information is already represented in
> the data schema for historical reasons. Since the schemas are not known in
> advance, `SELECT *` is the practical projection. Retaining the schema
> avoids a
> custom UDF solely for removing the extra operation column.
>
> There are two possible designs:
>
> 1. Add an optional TO_CHANGELOG parameter, for example
>    `include_op_column => FALSE`, to omit the operation column.
> 2. Keep the PTF API as it is and use the SQL projection that will become
>    available with the Calcite 1.42 upgrade:
>
>    SELECT * EXCLUDE(op)
>    FROM TO_CHANGELOG(input => TABLE source_table, ...);
>
> Omitting the operation column does not make a retract stream append-only in
> the usual sense. Callers that receive updates or deletes need an
> appropriate
> representation or filtering strategy; otherwise different change events can
> look identical after the operation code is removed. The motivating pipeline
> handles this separately.
>
> While `SELECT * EXCLUDE(op)` would solve this specific use case, I would
> prefer
> changing the PTF. Requiring TO_CHANGELOG to change the output schema feels
> like
> a sometimes unnecessary restriction. An option with a reasonable default
> adds
> little complexity, and most users do not even need to know about it, while
> it
> leaves more room for flexibility to retain the input schema with `SELECT *`
> when needed.
>
> WDYT?
>
> Best,
> Ferenc
>
> [1] https://github.com/apache/flink/pull/29102
>