Kafka Connect: Paused connector but still processing data

[Stephane Maarek <st...@simplemachines.com.au>]

Hi,

I have paused my connector yet it’s still very much active and processing
data. I can see because the offset lag keeps on decreasing (it got 100M
messages to read yet).
Is such a bug known? When I get the status it says PAUSED

(It is a custom connector, but I don’t think I’ve implemented anything in
an odd way)

Thanks!
Stephane

Re: Kafka Connect: Paused connector but still processing data

[Stephane Maarek <st...@simplemachines.com.au>]

Hi Ewen

I can't reproduce the bug but I was just using the standard rest API. I'll
let you know if it happens again once 0.10.2.0 is released or if I find a
reliable way to reproduce

Regards
Stephane

On 24 Jan. 2017 3:25 pm, "Ewen Cheslack-Postava" <ew...@confluent.io> wrote:

> There was this issue: https://issues.apache.org/jira/browse/KAFKA-4527
> which was a test failure that had to do with updating the status as soon as
> the request to pause the connector was received rather than after it was
> processed. The corresponding PR fixed that (and will be released in
> 0.10.2.0).
>
> If your connector is not properly responding to requests to stop, it's
> possible you would see the connector continue to process data incorrectly.
> If it is receiving the request to stop processing data (source connectors
> will be stopped, sink connectors will simply pause consumption), then it
> sounds like you found a way to trigger a bug. If you have a set of steps to
> reproduce the issue, a bug report would be appreciated.
>
> -Ewen
>
> On Mon, Jan 16, 2017 at 11:01 PM, Stephane Maarek <
> stephane@simplemachines.com.au> wrote:
>
> > Hi,
> >
> > I have paused my connector yet it’s still very much active and processing
> > data. I can see because the offset lag keeps on decreasing (it got 100M
> > messages to read yet).
> > Is such a bug known? When I get the status it says PAUSED
> >
> > (It is a custom connector, but I don’t think I’ve implemented anything in
> > an odd way)
> >
> > Thanks!
> > Stephane
> >
>

Re: Kafka Connect: Paused connector but still processing data

[Ewen Cheslack-Postava <ew...@confluent.io>]

There was this issue: https://issues.apache.org/jira/browse/KAFKA-4527
which was a test failure that had to do with updating the status as soon as
the request to pause the connector was received rather than after it was
processed. The corresponding PR fixed that (and will be released in
0.10.2.0).

If your connector is not properly responding to requests to stop, it's
possible you would see the connector continue to process data incorrectly.
If it is receiving the request to stop processing data (source connectors
will be stopped, sink connectors will simply pause consumption), then it
sounds like you found a way to trigger a bug. If you have a set of steps to
reproduce the issue, a bug report would be appreciated.

-Ewen

On Mon, Jan 16, 2017 at 11:01 PM, Stephane Maarek <
stephane@simplemachines.com.au> wrote:

> Hi,
>
> I have paused my connector yet it’s still very much active and processing
> data. I can see because the offset lag keeps on decreasing (it got 100M
> messages to read yet).
> Is such a bug known? When I get the status it says PAUSED
>
> (It is a custom connector, but I don’t think I’ve implemented anything in
> an odd way)
>
> Thanks!
> Stephane
>