[Discuss] Efficient CDC Upserts

[Ankit Kumar <26...@gmail.com>]

Hi all,

I was working with CDC use cases on Iceberg and came across this doc:
https://docs.google.com/document/d/1kyyJp4masbd1FrIKUHF1ED_z1hTARL8bNoKCgb7fhSQ/edit?tab=t.0
and thread here:
https://lists.apache.org/thread/6fhpjszsfxd8p0vfzc3k5vw7zmcyv2mq

It was a very interesting discussion that was happening, pointing out the
pain problem. Where the real-time query and the query on the mirror table
with respect to cost were the center point of the discussion.

Is there any conclusion that has been reached on this topic? (Some
solutions were discussed in the thread by @russel and by @Jack Ye)

My concerns are that some catalogs do not support branching as of now, as
well as Iceberg logical views (for real-time purposes).


Thanks

Re: [Discuss] Efficient CDC Upserts

[Péter Váry <pe...@gmail.com>]

Hi Ankit,

You might want to take a look at the Iceberg Secondary Index proposal too:

   - Index metadata:
   https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0
   - Key lookup index:
   https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8

Having an available key lookup index could help writing upserts immediately.
We are planning to migrate Flink to the new indexes once they are
available. The ongoing work could be checked here:

   - Proposed design:
   https://docs.google.com/document/d/1Jz4Fjt-6jRmwqbgHX_u0ohuyTB9ytDzfslS7lYraIjk/edit?tab=t.5qlfbj9bk8i3
   - Mailing list discussion:
   https://lists.apache.org/thread/gldqrycmo24r0vo77wzcj9s9b04rtvy7
   - First step: Flink 2.1: Add ConvertEqualityDeletes maintenance task -
   https://github.com/apache/iceberg/pull/15996

I hope this helps.
Thanks,
Peter

Ankit Kumar <26...@gmail.com> ezt írta (időpont: 2026. jún. 2.,
K, 15:05):

> Hi all,
>
> I was working with CDC use cases on Iceberg and came across this doc:
> https://docs.google.com/document/d/1kyyJp4masbd1FrIKUHF1ED_z1hTARL8bNoKCgb7fhSQ/edit?tab=t.0
> and thread here:
> https://lists.apache.org/thread/6fhpjszsfxd8p0vfzc3k5vw7zmcyv2mq
>
> It was a very interesting discussion that was happening, pointing out the
> pain problem. Where the real-time query and the query on the mirror table
> with respect to cost were the center point of the discussion.
>
> Is there any conclusion that has been reached on this topic? (Some
> solutions were discussed in the thread by @russel and by @Jack Ye)
>
> My concerns are that some catalogs do not support branching as of now, as
> well as Iceberg logical views (for real-time purposes).
>
>
> Thanks
>