[DISCUSS] Kafka Connect delta writer support

[ismail simsek <is...@gmail.com>]

Hi All,

I have ported the Kafka Connect delta writer feature, however there are
some questions around should we support it or wait for the new solution for
Equality deletes.

And there is some interest from the community to have it, so opening this
threat to discuss it.

PR https://github.com/apache/iceberg/pull/12070

I appreciate any thoughts or ideas from the community.

Best
ismail

Re: [DISCUSS] Kafka Connect delta writer support

[Bryan Keller <br...@gmail.com>]

The Flink sink contributors are working on a solution for streaming deletes without using equality deletes. I'm hoping we can implement a similar solution in the Kafka Connect sink.

-Bryan

> On Jun 10, 2025, at 9:37 AM, Ryan Blue <rd...@gmail.com> wrote:
> 
> I'm strongly against writing equality deletes from the KC writer because it can't sort to make the deletes more efficient to apply. I don't think that equality deletes should be used in situations like this and that it is only going to cause pain for users that don't understand that they need offline maintenance.
> 
> On Tue, Jun 10, 2025 at 2:03 AM ismail simsek <ismailxsimsek@gmail.com <ma...@gmail.com>> wrote:
>> Hi All,
>> 
>> I have ported the Kafka Connect delta writer feature, however there are some questions around should we support it or wait for the new solution for Equality deletes.
>> 
>> And there is some interest from the community to have it, so opening this threat to discuss it. 
>> 
>> PR https://github.com/apache/iceberg/pull/12070 
>> 
>> I appreciate any thoughts or ideas from the community.
>> 
>> Best 
>> ismail

Re: [DISCUSS] Kafka Connect delta writer support

[Ryan Blue <rd...@gmail.com>]

I'm strongly against writing equality deletes from the KC writer because it
can't sort to make the deletes more efficient to apply. I don't think that
equality deletes should be used in situations like this and that it is only
going to cause pain for users that don't understand that they need offline
maintenance.

On Tue, Jun 10, 2025 at 2:03 AM ismail simsek <is...@gmail.com>
wrote:

> Hi All,
>
> I have ported the Kafka Connect delta writer feature, however there are
> some questions around should we support it or wait for the new solution for
> Equality deletes.
>
> And there is some interest from the community to have it, so opening this
> threat to discuss it.
>
> PR https://github.com/apache/iceberg/pull/12070
>
> I appreciate any thoughts or ideas from the community.
>
> Best
> ismail
>