Re: [DISCUSS] KIP-762 Delete Committed Connect Records

[Mickael Maison <mi...@gmail.com>]

Hi Ryanne,

Thanks for the KIP, this is an interesting use case.

I agree it's best to have the runtime take care of the deletion.
However, I wonder if it would make more sense to have the
configuration on connectors instead of the runtime.

While Workers perform the deletion, it's very much driven by
connectors. It's true that when having multiple sink connectors that
use the same topics, you'd need to enable deletion on each but using
connector configuration looks like a more natural way to defining how
a pipeline should work.
Also if configured on the runtime, each time a new pipeline is setup,
it may require reconfiguring and restarting the Connect cluster.

What do you think?

On Sat, Jul 17, 2021 at 3:16 PM Ryanne Dolan <ry...@gmail.com> wrote:
>
> Whoops, looks like I got the KIP number wrong in the original email subject
> line. Please use this corrected thread.
>
> Ryanne
>
> On Fri, Jul 16, 2021, 3:45 PM Ryanne Dolan <ry...@gmail.com> wrote:
>
> > Hey y'all, please review the following small proposal:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-762%3A+Delete+Committed+Connect+Records
> >
> > The idea is to enable Source->Sink pipelines to clean up after themselves
> > by automatically truncating intermediate topics.
> >
> > Ryanne
> >
> >

Re: [DISCUSS] KIP-762 Delete Committed Connect Records

[Ryanne Dolan <ry...@gmail.com>]

Mickael, thanks for taking a look. I wouldn't be opposed to making this a
per-connector configuration. I guess each topic-partition is consumed by
one WorkerSinkTask, so it should be safe for the worker to truncate such
partitions if all associated Connectors opt-in.

I think in that case it may make sense to remove the regex configuration
part, which appeals to me.

Thanks for the suggestion.

Ryanne

On Thu, Aug 19, 2021, 9:18 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Ryanne,
>
> Thanks for the KIP, this is an interesting use case.
>
> I agree it's best to have the runtime take care of the deletion.
> However, I wonder if it would make more sense to have the
> configuration on connectors instead of the runtime.
>
> While Workers perform the deletion, it's very much driven by
> connectors. It's true that when having multiple sink connectors that
> use the same topics, you'd need to enable deletion on each but using
> connector configuration looks like a more natural way to defining how
> a pipeline should work.
> Also if configured on the runtime, each time a new pipeline is setup,
> it may require reconfiguring and restarting the Connect cluster.
>
> What do you think?
>
> On Sat, Jul 17, 2021 at 3:16 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > Whoops, looks like I got the KIP number wrong in the original email
> subject
> > line. Please use this corrected thread.
> >
> > Ryanne
> >
> > On Fri, Jul 16, 2021, 3:45 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > > Hey y'all, please review the following small proposal:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-762%3A+Delete+Committed+Connect+Records
> > >
> > > The idea is to enable Source->Sink pipelines to clean up after
> themselves
> > > by automatically truncating intermediate topics.
> > >
> > > Ryanne
> > >
> > >
>