You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mazhar Shaikh <ma...@gmail.com> on 2015/12/03 16:08:52 UTC
Error in handling geo redundant kafka process
Hello All,
I'm using "librdkafka" for my C Project which has is needed to support geo
redundant kafka process (zookeeper + Broker).
Machine 1 : Producer1 : Broker IP"sysctrl1.vsepx.broker.com:9092,
sysctrl2.vsepx.broker.com:9092"
Machine 2 : Broker1 : advertised.host.name=sysctrl1.vsepx.broker.com
Machine 3 : Broker2 : advertised.host.name=sysctrl2.vsepx.broker.com
Scenario:
1. Broker1 is UP, Broker 2 is Down.
2. Producer1 is sending messages to Broker1 successfully
3. Broker1 goes down.
4. Broker2 comes UP
5. Producer1 connection with broker 1 fails, But Connection with
Broker2 is established.
6. Producer1 exchanges meta data with Broker2, But messages are not
being sent to Broker2, Instead the producer1 is still trying to send the
same to Broker1 (which is down).
7. All Messages Fail.
Now, If I kill and restart Producer1, then all the new messages are
succesfully sent to Broker2.
Broker logs:
[2015-12-03 09:28:24,573] INFO New leader is 0
(kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2015-12-03 09:28:24,579] INFO Registered broker 0 at path /brokers/ids/0
with address sysctrl2.vsepx.broker.com:9092. (kafka.utils.ZkUtils$)
[2015-12-03 09:28:24,593] INFO [Kafka Server 0], started
(kafka.server.KafkaServer)
[2015-12-03 09:28:24,718] INFO [ReplicaFetcherManager on broker 0] Removed
fetcher for partitions
[topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],
topic1,14
<https://github.com/edenhill/librdkafka/issues/kafka.server.ReplicaFetcherManager>
[2015-12-03 09:28:24,982] INFO [ReplicaFetcherManager on broker 0] Removed
fetcher for partitions
[topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],
topic1,14
<https://github.com/edenhill/librdkafka/issues/kafka.server.ReplicaFetcherManager>
[2015-12-03 09:30:10,195] INFO Closing socket connection to /12.1.1.81.
(kafka.network.Processor)
Below are few logs from sample stub code
% Sent 3 bytes to topic topic1 partition -1, outQlen[127]
log_cb : rdkafka#producer-0: [7] [RECV] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Received MetadataResponse (473
bytes, CorrId 12, rtt 142.35ms)]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: ===== Received metadata from
sysctrl2.vsepx.broker.com:9092/bootstrap =====]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: 1 brokers, 1 topics]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Broker #0/1:
sysctrl2.vsepx.broker.com:9092 NodeId 0]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic #0/1: topic1 with 16
partitions]
log_cb : rdkafka#producer-0: [7] [PARTCNT] [No change in partition count
for topic topic1]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 8 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [8] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 11 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [11] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 2 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [2] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 5 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [5] with 8
messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 14 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [14] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 4 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [4] with 8
messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 13 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [13] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 7 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [7] with 7
messages (17 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 1 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [1] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 10 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [10] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 9 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [9] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 3 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [3] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 12 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [12] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 15 Leader
0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [15] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 6 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [6] with 8
messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [
sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 0 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [0] with 8
messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [PARTCNT] [Partitioning 0 unassigned
messages in topic topic1 to 16 partitions]
log_cb : rdkafka#producer-0: [7] [UAS] [0/0 messages were partitioned in
topic topic1]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0:
broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [
sysctrl1.vsepx.broker.com:9092/bootstrap: broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [
sysctrl1.vsepx.broker.com:9092/bootstrap: couldn't connect to ipv4#
192.168.108.140:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [
sysctrl1.vsepx.broker.com:9092/bootstrap: failed: err: Local: Communication
failure with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [
sysctrl1.vsepx.broker.com:9092/bootstrap: Purging bufq with 0 buffers]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0:
couldn't connect to ipv4#192.168.108.140:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [
sysctrl1.vsepx.broker.com:9092/0: failed: err: Local: Communication failure
with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/0:
Purging bufq with 0 buffers]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0:
Undelegating topic1 [0]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker
sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [0]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic
topic1 [0]]
Regards,
Mazhar