You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mathieu Fenniak <ma...@replicon.com> on 2017/01/13 18:32:23 UTC
Kafka Connect requestTaskReconfiguration
Hey kafka-users,
Is it normal for a Kafka Connect source connector that
calls requestTaskReconfiguration to cause all the connectors on the
kafka-connect distributed system to be stopped and started?
One of my three connectors (2x source, 1x sink) runs a background thread
that will occasionally invoke `context.requestTaskReconfiguration()` when
it detects new work that it wants to distribute to its tasks. When this
occurs, I observe all three of the connectors stop and start. One of the
other connectors doesn't have the smoothest stop/start cycle, so I'm hoping
that this might be avoidable?
I'm running on Kafka Connect 0.10.1.0.
Mathieu
Re: Kafka Connect requestTaskReconfiguration
Posted by Mathieu Fenniak <ma...@replicon.com>.
OK, thanks Ewen.
On Sun, Jan 15, 2017 at 4:42 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> This is currently expected. Internally the Connect cluster uses the same
> rebalancing process as consumer groups which means it has similar
> limitations -- all tasks must stop just as you would need to stop consuming
> from all partitions and commit offsets during a consumer group rebalance.
>
> There's an issue filed about this: https://issues.apache.org/
> jira/browse/KAFKA-3351 and it's something we're aware eventually becomes a
> scalability limitations. There are some ideas about how to avoid this, but
> nothing concrete on the roadmap yet.
>
> -Ewen
>
> On Fri, Jan 13, 2017 at 10:32 AM, Mathieu Fenniak <
> mathieu.fenniak@replicon.com> wrote:
>
> > Hey kafka-users,
> >
> > Is it normal for a Kafka Connect source connector that
> > calls requestTaskReconfiguration to cause all the connectors on the
> > kafka-connect distributed system to be stopped and started?
> >
> > One of my three connectors (2x source, 1x sink) runs a background thread
> > that will occasionally invoke `context.requestTaskReconfiguration()`
> when
> > it detects new work that it wants to distribute to its tasks. When this
> > occurs, I observe all three of the connectors stop and start. One of the
> > other connectors doesn't have the smoothest stop/start cycle, so I'm
> hoping
> > that this might be avoidable?
> >
> > I'm running on Kafka Connect 0.10.1.0.
> >
> > Mathieu
> >
>
Re: Kafka Connect requestTaskReconfiguration
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
This is currently expected. Internally the Connect cluster uses the same
rebalancing process as consumer groups which means it has similar
limitations -- all tasks must stop just as you would need to stop consuming
from all partitions and commit offsets during a consumer group rebalance.
There's an issue filed about this: https://issues.apache.org/
jira/browse/KAFKA-3351 and it's something we're aware eventually becomes a
scalability limitations. There are some ideas about how to avoid this, but
nothing concrete on the roadmap yet.
-Ewen
On Fri, Jan 13, 2017 at 10:32 AM, Mathieu Fenniak <
mathieu.fenniak@replicon.com> wrote:
> Hey kafka-users,
>
> Is it normal for a Kafka Connect source connector that
> calls requestTaskReconfiguration to cause all the connectors on the
> kafka-connect distributed system to be stopped and started?
>
> One of my three connectors (2x source, 1x sink) runs a background thread
> that will occasionally invoke `context.requestTaskReconfiguration()` when
> it detects new work that it wants to distribute to its tasks. When this
> occurs, I observe all three of the connectors stop and start. One of the
> other connectors doesn't have the smoothest stop/start cycle, so I'm hoping
> that this might be avoidable?
>
> I'm running on Kafka Connect 0.10.1.0.
>
> Mathieu
>