You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Abhisek Saikia (JIRA)" <ji...@apache.org> on 2017/05/23 18:46:04 UTC
[jira] [Comment Edited] (KAFKA-5296) Unable to write to some
partitions of newly created topic in 10.2
[ https://issues.apache.org/jira/browse/KAFKA-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021577#comment-16021577 ]
Abhisek Saikia edited comment on KAFKA-5296 at 5/23/17 6:45 PM:
----------------------------------------------------------------
Any new topic creation is running into issues now. Here are the steps I am using to reproduce the issue-
1. Create a new topic with 20 partitions and 3 replicas -
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 20 --topic testcreate2
2. Get the topic details using kafka-topics command-
kafka-topics.sh --describe --zookeeper localhost:2181 --topic testcreate2
Topic:testcreate2 PartitionCount:20 ReplicationFactor:3 Configs:
Topic: testcreate2 Partition: 0 Leader: 138 Replicas: 138,132,133 Isr: 138,132,133
Topic: testcreate2 Partition: 1 Leader: 139 Replicas: 139,133,135 Isr: 139,133,135
Topic: testcreate2 Partition: 2 Leader: 140 Replicas: 140,135,136 Isr: 140,135,136
Topic: testcreate2 Partition: 3 Leader: 130 Replicas: 130,136,137 Isr: 130,136,137
Topic: testcreate2 Partition: 4 Leader: 131 Replicas: 131,137,138 Isr: 131,137,138
Topic: testcreate2 Partition: 5 Leader: 132 Replicas: 132,138,139 Isr: 132,138,139
Topic: testcreate2 Partition: 6 Leader: 133 Replicas: 133,139,140 Isr: 133,139,140
Topic: testcreate2 Partition: 7 Leader: 135 Replicas: 135,140,130 Isr: 135,140,130
Topic: testcreate2 Partition: 8 Leader: 136 Replicas: 136,130,131 Isr: 136,130,131
Topic: testcreate2 Partition: 9 Leader: 137 Replicas: 137,131,132 Isr: 137,131,132
Topic: testcreate2 Partition: 10 Leader: 138 Replicas: 138,133,135 Isr: 138,133,135
Topic: testcreate2 Partition: 11 Leader: 139 Replicas: 139,135,136 Isr: 139,135,136
Topic: testcreate2 Partition: 12 Leader: 140 Replicas: 140,136,137 Isr: 140,136,137
Topic: testcreate2 Partition: 13 Leader: 130 Replicas: 130,137,138 Isr: 130,137,138
Topic: testcreate2 Partition: 14 Leader: 131 Replicas: 131,138,139 Isr: 131,138,139
Topic: testcreate2 Partition: 15 Leader: 132 Replicas: 132,139,140 Isr: 132,139,140
Topic: testcreate2 Partition: 16 Leader: 133 Replicas: 133,140,130 Isr: 133,140,130
Topic: testcreate2 Partition: 17 Leader: 135 Replicas: 135,130,131 Isr: 135,130,131
Topic: testcreate2 Partition: 18 Leader: 136 Replicas: 136,131,132 Isr: 136,131,132
Topic: testcreate2 Partition: 19 Leader: 137 Replicas: 137,132,133 Isr: 137,132,133
3. Using Simple consumer try to read the topic metadata for this topic. It seems topic metadata is found for only 11 partitions out of 20.
{
"messages": [
{
"topicMetaDataList": [
{
"topic": "testcreate2",
"paritionMetaData": [
{
"partition": 0,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 132,
"port": 9092
},
{
"id": 133,
"port": 9092
}
]
},
{
"partition": 1,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 18,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 3,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 136,
"port": 9092
},
{
"id": 137,
"port": 9092
}
]
},
{
"partition": 4,
"leader": {
"id": 131,
"port": 9092
},
"isr": [
{
"id": 131,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
},
{
"partition": 5,
"leader": {
"id": 132,
"port": 9092
},
"isr": [
{
"id": 132,
"port": 9092
},
{
"id": 138,
"port": 9092
},
{
"id": 139,
"port": 9092
}
]
},
{
"partition": 8,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 130,
"port": 9092
},
{
"id": 131,
"port": 9092
}
]
},
{
"partition": 9,
"leader": {
"id": 137,
"port": 9092
},
"isr": [
{
"id": 137,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 10,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 11,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 135,
"port": 9092
},
{
"id": 136,
"port": 9092
}
]
},
{
"partition": 13,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
}
]
}
]
}
]
}
4. Above payload does not contain metadata for partition 14 but as per kafka-topics.sh command , broker 131 is supposed to be the leader broker for this partition.
The log directory of this broker does not have any folder with name "testcreate2-2" . It has the following folders -
testcreate2-18:
00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
testcreate2-4:
00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
testcreate2-8:
00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
testcreate2-9:
00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
As per the following logs of broker 131, its receiving LeaderAndIsr request from multiple controller for different paritions.
./state-change.log.2017-05-22-18:[2017-05-22 18:48:30,108] TRACE Broker 131 received LeaderAndIsr request PartitionState(controllerEpoch=64, leader=131, leaderEpoch=0, isr=[131, 138, 139], zkVersion=0, replicas=[139, 131, 138]) correlation id 5453 from controller 130 epoch 64 for partition [testcreate2,14] (state.change.logger)
./state-change.log.2017-05-22-18:[2017-05-22 18:48:30,106] TRACE Broker 131 received LeaderAndIsr request PartitionState(controllerEpoch=70, leader=136, leaderEpoch=0, isr=[136, 131, 132], zkVersion=0, replicas=[132, 136, 131]) correlation id 814 from controller 137 epoch 70 for partition [testcreate2,18] (state.change.logger)
was (Author: asaikia):
Any new topic creation is running into issues now. Here are the steps I am using to reproduce the issue-
1. Create a new topic with 20 partitions and 3 replicas -
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 20 --topic testcreate2
2. Get the topic details using kafka-topics command-
kafka-topics.sh --describe --zookeeper localhost:2181 --topic testcreate2
Topic:testcreate2 PartitionCount:20 ReplicationFactor:3 Configs:
Topic: testcreate2 Partition: 0 Leader: 138 Replicas: 138,132,133 Isr: 138,132,133
Topic: testcreate2 Partition: 1 Leader: 139 Replicas: 139,133,135 Isr: 139,133,135
Topic: testcreate2 Partition: 2 Leader: 140 Replicas: 140,135,136 Isr: 140,135,136
Topic: testcreate2 Partition: 3 Leader: 130 Replicas: 130,136,137 Isr: 130,136,137
Topic: testcreate2 Partition: 4 Leader: 131 Replicas: 131,137,138 Isr: 131,137,138
Topic: testcreate2 Partition: 5 Leader: 132 Replicas: 132,138,139 Isr: 132,138,139
Topic: testcreate2 Partition: 6 Leader: 133 Replicas: 133,139,140 Isr: 133,139,140
Topic: testcreate2 Partition: 7 Leader: 135 Replicas: 135,140,130 Isr: 135,140,130
Topic: testcreate2 Partition: 8 Leader: 136 Replicas: 136,130,131 Isr: 136,130,131
Topic: testcreate2 Partition: 9 Leader: 137 Replicas: 137,131,132 Isr: 137,131,132
Topic: testcreate2 Partition: 10 Leader: 138 Replicas: 138,133,135 Isr: 138,133,135
Topic: testcreate2 Partition: 11 Leader: 139 Replicas: 139,135,136 Isr: 139,135,136
Topic: testcreate2 Partition: 12 Leader: 140 Replicas: 140,136,137 Isr: 140,136,137
Topic: testcreate2 Partition: 13 Leader: 130 Replicas: 130,137,138 Isr: 130,137,138
Topic: testcreate2 Partition: 14 Leader: 131 Replicas: 131,138,139 Isr: 131,138,139
Topic: testcreate2 Partition: 15 Leader: 132 Replicas: 132,139,140 Isr: 132,139,140
Topic: testcreate2 Partition: 16 Leader: 133 Replicas: 133,140,130 Isr: 133,140,130
Topic: testcreate2 Partition: 17 Leader: 135 Replicas: 135,130,131 Isr: 135,130,131
Topic: testcreate2 Partition: 18 Leader: 136 Replicas: 136,131,132 Isr: 136,131,132
Topic: testcreate2 Partition: 19 Leader: 137 Replicas: 137,132,133 Isr: 137,132,133
3. Using Simple consumer try to read the topic metadata for this topic. It seems topic metadata is found for only 11 partitions out of 20.
{
"messages": [
{
"topicMetaDataList": [
{
"topic": "testcreate2",
"paritionMetaData": [
{
"partition": 0,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 132,
"port": 9092
},
{
"id": 133,
"port": 9092
}
]
},
{
"partition": 1,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 18,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 3,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 136,
"port": 9092
},
{
"id": 137,
"port": 9092
}
]
},
{
"partition": 4,
"leader": {
"id": 131,
"port": 9092
},
"isr": [
{
"id": 131,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
},
{
"partition": 5,
"leader": {
"id": 132,
"port": 9092
},
"isr": [
{
"id": 132,
"port": 9092
},
{
"id": 138,
"port": 9092
},
{
"id": 139,
"port": 9092
}
]
},
{
"partition": 8,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 130,
"port": 9092
},
{
"id": 131,
"port": 9092
}
]
},
{
"partition": 9,
"leader": {
"id": 137,
"port": 9092
},
"isr": [
{
"id": 137,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 10,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 11,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 135,
"port": 9092
},
{
"id": 136,
"port": 9092
}
]
},
{
"partition": 13,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
}
]
}
]
}
]
}
4. Above payload does not contain metadata for partition 2 but as per kafka-topics.sh command , broker 140 is supposed to be the leader broker for this partition.
The log directory of this broker does not have any folder with name "testcreate2-2"
> Unable to write to some partitions of newly created topic in 10.2
> -----------------------------------------------------------------
>
> Key: KAFKA-5296
> URL: https://issues.apache.org/jira/browse/KAFKA-5296
> Project: Kafka
> Issue Type: Bug
> Reporter: Abhisek Saikia
>
> We are using kafka 10.2 and the cluster was running fine for a month with 50 topics and now we are having issue in producing message by creating new topics. The create topic command is successful but producers are throwing error while writing to some partitions.
> Error in producer-
> java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for [topic1]-8: 30039 ms has passed since batch creation plus linger time
> at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70) ~[kafka-clients-0.10.2.0.jar:na]
> at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57) ~[kafka-clients-0.10.2.0.jar:na]
> at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) ~[kafka-clients-0.10.2.0.jar:na]
> On the broker side, I don't see any topic-parition folder getting created for the broker who is the leader for the partition.
> While using 0.8 client, the write used to hang while it starts writing to the partition having this issue. With 10.2 it resolved the the producer hang issue
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)