You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/02/14 20:59:00 UTC
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for
connector causes deleted tasks to show as UNASSIGNED
[ https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037293#comment-17037293 ]
ASF GitHub Bot commented on KAFKA-9472:
---------------------------------------
C0urante commented on pull request #8118: KAFKA-9472: Remove deleted tasks from status store
URL: https://github.com/apache/kafka/pull/8118
[Jira](https://issues.apache.org/jira/browse/KAFKA-9472)
Although the statuses for tasks are removed from the status store when their _connector_ is deleted, their status are not removed when only the task is deleted, which happens in the case that the number of tasks for a connector is reduced.
These changes add logic for deleting the statuses for those tasks from the status store whenever a rebalance has completed and the leader of a distributed cluster has detected that there are recently-deleted tasks. Standalone is also updated to accomplish this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2, 2.4.0, 2.3.1
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole numbers and {{n}} is strictly less than {{t1}}), the connector should then list {{t1 - n}} total tasks in its status (which can be queried via the {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in its status, with {{n}} of them being listed as {{UNASSIGNED}} and the remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when its connector is deleted. See relevant code snippets from the [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192] and [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520] classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)