[jira] [Commented] (KAFKA-13392) Timeout Exception triggering reassign partitions with --bootstrap-server option

["Jonas Lundholm Bertelsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719274#comment-17719274 ] 

Jonas Lundholm Bertelsen commented on KAFKA-13392:
--------------------------------------------------

We have seen this same issue with a Kafka 3.4 cluster where one broker is down and we want to reassign replicas away from it.

The `–zookeeper` argument is no longer available, so we were not able to use that workaround.

An alternative workaround is to simply omit the throttle parameter. This obviously has its own drawbacks. In our case we split the reassignment file into smaller parts and applied them one after the other to try and put a "manual throttle" on activity spikes.

> Timeout Exception triggering reassign partitions with --bootstrap-server option
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-13392
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13392
>             Project: Kafka
>          Issue Type: Bug
>          Components: admin
>    Affects Versions: 2.8.0
>            Reporter: Yevgeniy Korin
>            Priority: Minor
>
> *Scenario when we faced with this issue:*
>  One of three brokers is down. Add another (fourth) broker and try to reassign partitions using '--bootstrap-server'
>  option.
> *What's failed:*
> {code:java}
> /opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server xxx.xxx.xxx.xxx:9092 --reassignment-json-file /tmp/reassignment-20211021130718.json --throttle 100000000 --execute{code}
> failed with
> {code:java}
> Error: org.apache.kafka.common.errors.TimeoutException: Call(callName=incrementalAlterConfigs, deadlineMs=1634811369255, tries=1, nextAllowedTryMs=1634811369356) timed out at 1634811369256 after 1 attempt(s)
>  java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=incrementalAlterConfigs, deadlineMs=1634811369255, tries=1, nextAllowedTryMs=1634811369356) timed out at 1634811369256 after 1 attempt(s)
>  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
>  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
>  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
>  at kafka.admin.ReassignPartitionsCommand$.modifyInterBrokerThrottle(ReassignPartitionsCommand.scala:1435)
>  at kafka.admin.ReassignPartitionsCommand$.modifyReassignmentThrottle(ReassignPartitionsCommand.scala:1412)
>  at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:974)
>  at kafka.admin.ReassignPartitionsCommand$.handleAction(ReassignPartitionsCommand.scala:255)
>  at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:216)
>  at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>  Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=incrementalAlterConfigs, deadlineMs=1634811369255, tries=1, nextAllowedTryMs=1634811369356) timed out at 1634811369256 after 1 attempt(s)
>  Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: incrementalAlterConfigs{code}
>  *Expected behavio**:*
>  partition reassignment process started.
> *Workaround:*
>  Trigger partition reassignment process using '--zookeeper' option:
> {code:java}
> /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper zookeeper.my.company:2181/kafka-cluster --reassignment-json-file /tmp/reassignment-20211021130718.json --throttle 100000000 --execute{code}
>  *Additional info:*
>  We are able to trigger partition reassignment using '--bootstrap-server' option with no exceptions when all four brokers are alive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)