[DISCUSS] KIP-318: Make Kafka Connect Source idempotent

[Stephane Maarek <st...@simplemachines.com.au>]

KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent


By looking at the code, it seems Worker.java is where the magic happens,
but do we need to also operate changes to KafkaBasedLog.java (~line 241) ?

Love to hear your thoughts!

Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

[Stephane Maarek <st...@simplemachines.com.au>]

Hi Randall

Thanks for your feedback

1) user can override: yes they can but they most surely won't know about
it. I didn't know about this improvement until I got on twitter and
exchanged with Ismael. I would say most users don't even know Kafka connect
is running with acks=all. My understanding behind the philosophy of Kafka
connect was that users only worry about writing a connector and the
framework makes the whole ETL safe. In that regards I think it's important
to increase the level of safety by preventing network duplicates (I don't
think anyone is against not having duplicates) and at the same time
increasing performance by having more in flight requests while keeping
ordering guarantees (I don't think anyone is against that either). So the
behaviour changes but I don't see any drawbacks to it.

1*) I'm very much allergic to introducing more Configs, but if the
community desires we can control that behaviour explicitly with a new
config and default the behaviour to false. It would give the users an
easier opt in and eventually we'll flip the config default to true


2) is there an easy way to clearly detect if a broker is running a specific
version of the API. If so, I don't mind including an if statement for a
conditional worker configuration and that would solve backwards
compatibility?

Thanks
Stephane




On Wed., 20 Jun. 2018, 10:54 pm Randall Hauch, <rh...@gmail.com> wrote:

> Thanks for starting this conversation, Stephane. I have a few questions.
>
> The worker already accepts nearly all producer properties already, and all
> `producer.*` properties override any hard-coded properties defined in
> `Worker.java`. So isn't it currently possible for a user to define these
> properties in their worker configuration if they want?
>
> Second, wouldn't this change the default behavior for existing worker
> configurations that have not overridden these properties? IOW, we would
> need to address the migration path to ensure backward compatibility.
>
> Third, the KIP mentions but does not really address the problem of running
> workers against pre-1.0 Kafka clusters. That definitely is something that
> happens frequently, so what is the planned approach for addressing this
> compatibility concern?
>
> All of these factors are likely why this has not yet been addressed to
> date: it's already possible for users to enable this feature, but doing it
> by default has compatibility concerns.
>
> Thoughts?
>
> Best regards,
>
> Randall
>
>
> On Wed, Jun 20, 2018 at 1:17 AM, Stephane Maarek <
> stephane@simplemachines.com.au> wrote:
>
> > KIP link:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 318%3A+Make+Kafka+Connect+Source+idempotent
> >
> >
> > By looking at the code, it seems Worker.java is where the magic happens,
> > but do we need to also operate changes to KafkaBasedLog.java (~line 241)
> ?
> >
> > Love to hear your thoughts!
> >
>

Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

[Randall Hauch <rh...@gmail.com>]

Thanks for starting this conversation, Stephane. I have a few questions.

The worker already accepts nearly all producer properties already, and all
`producer.*` properties override any hard-coded properties defined in
`Worker.java`. So isn't it currently possible for a user to define these
properties in their worker configuration if they want?

Second, wouldn't this change the default behavior for existing worker
configurations that have not overridden these properties? IOW, we would
need to address the migration path to ensure backward compatibility.

Third, the KIP mentions but does not really address the problem of running
workers against pre-1.0 Kafka clusters. That definitely is something that
happens frequently, so what is the planned approach for addressing this
compatibility concern?

All of these factors are likely why this has not yet been addressed to
date: it's already possible for users to enable this feature, but doing it
by default has compatibility concerns.

Thoughts?

Best regards,

Randall


On Wed, Jun 20, 2018 at 1:17 AM, Stephane Maarek <
stephane@simplemachines.com.au> wrote:

> KIP link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 318%3A+Make+Kafka+Connect+Source+idempotent
>
>
> By looking at the code, it seems Worker.java is where the magic happens,
> but do we need to also operate changes to KafkaBasedLog.java (~line 241) ?
>
> Love to hear your thoughts!
>