You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Seshadri, Balaji" <Ba...@dish.com> on 2014/09/24 19:48:04 UTC
BadVersion state in Kafka Logs
Hi,
We got the below error in our logs and our consumers stopped consuming any data ?.It worked only after restart.
We would like to confirm that it's because we are running with 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
Please let me know if it's this KAFKA-1382 causing the issue.
Thanks,
Balaji
From: Gulia, Vikram
Sent: Wednesday, September 24, 2014 8:43 AM
To: Sharma, Navdeep; #IT-MAD DES; #IT-MAA
Cc: Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Adding full MAA distro.
DES Offshore looked in to the logs on kafka servers and seems like the issue we encountered yesterday may be described in these threads, please have a look -
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the fix/patch which is available in 0.8.1.2/0.8.2)
Thank You,
Vikram Gulia
From: Sharma, Navdeep
Sent: Wednesday, September 24, 2014 6:53 AM
To: Gulia, Vikram; #IT-MAD DES
Cc: #IT-MAA Offshore; Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Hi Vikram,
We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found that the error occurred only on 23 Sept. This is not historical as we checked last 4 days logs.
It looks like that consumer got stopped on September 22 2014 for Linux patching activty.MAA started consumer September 23 2014 at 1:00 AM.
Issue in server log " BadVersion for /brokers/topics/rain-burn-in/partitions/121/state" but it is not present in previous 4 days logs.
More detail of this error can be found at-
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
We are not sure about data loss in this scenario and working on this.
[cid:image001.png@01CFD7D3.80A9B2B0]
[cid:image002.png@01CFD7D3.80A9B2B0]
Let us know if any concerns.
[cid:image003.gif@01CFD7D3.80A9B2B0]
Navdeep Sharma
Developer - offshore, Middleware Applications & Development
o India: 0120-4532000 - 2234
c: +91-9911698102
From: Gulia, Vikram
Sent: Tuesday, September 23, 2014 6:17 PM
To: #IT-MAD DES
Subject: FW: 9/23 prod issue - offline kafka partitions.
DES Offshore dev,
Please work with MAA offshore to monitor the kafka broker as we had this incident where lot of partitions went offline around 1.45 PM MST and MAA has to restart the kafka servers. We may have lost messages and we need to see if there is a way to figure out what was the impact.
Also, check the logs for kafka servers and see if we can figure out why did partitions go offline or are un-available? Let us know if you find anything relevant.
Thank You,
Vikram Gulia
From: Nielsen, Andy
Sent: Tuesday, September 23, 2014 5:04 PM
To: #IT-MAD DES; Gulia, Vikram
Cc: #IT-MAA
Subject: 9/23 prod issue - offline kafka partitions.
desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
topic: account-access partition: 21 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 51 leader: -1 replicas: 4,6,1 isr:
topic: account-access partition: 81 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 111 leader: -1 replicas: 4,6,1 isr: 1
topic: account-activated partition: 13 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 43 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 73 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 103 leader: -1 replicas: 4,6,1 isr: 1
topic: account-adjustment-issued partition: 27 leader: -1 replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 87 leader: -1 replicas: 4,6,1 isr: 1
topic: account-adjustment-issued partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 11 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 41 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 71 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 101 leader: -1 replicas: 4,6,1 isr:
topic: account-info-updated partition: 7 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 37 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 67 leader: -1 replicas: 4,6,1 isr:
topic: account-info-updated partition: 97 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 127 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 21 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 51 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 81 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 111 leader: -1 replicas: 4,6,1 isr: 1
topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 29 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 23 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 53 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 83 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 113 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 32 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 122 leader: -1 replicas: 4,6,1 isr: 1
topic: leads-service partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 84 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1 isr: 1
topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 21 leader: -1 replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 51 leader: -1 replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 81 leader: -1 replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 111 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 25 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 55 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 85 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 115 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 8 leader: -1 replicas: 4,6,1 isr: 1
topic: order-modified partition: 38 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 98 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 128 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 27 leader: -1 replicas: 4,6,1 isr: 1
topic: order-response partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 87 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: outbound-call-attempted partition: 13 leader: -1 replicas: 4,6,1 isr:
topic: outbound-call-attempted partition: 43 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-call-attempted partition: 73 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-call-attempted partition: 103 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 4 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 34 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 64 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 94 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-communications partition: 124 leader: -1 replicas: 4,6,1 isr: 1
topic: postal-mail-undeliverable partition: 15 leader: -1 replicas: 4,6,1 isr: 1
topic: postal-mail-undeliverable partition: 45 leader: -1 replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 75 leader: -1 replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 105 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1 isr: 1
topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 14 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 44 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 74 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 104 leader: -1 replicas: 4,6,1 isr: 1
Andy Nielsen
Middleware Application Admin
303-723-2347
cell:720-971-2856
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
Please find the log attached.
From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
Sent: Wednesday, September 24, 2014 11:48 AM
To: 'users@kafka.apache.org'
Subject: BadVersion state in Kafka Logs
Hi,
We got the below error in our logs and our consumers stopped consuming any data ?.It worked only after restart.
We would like to confirm that it's because we are running with 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
Please let me know if it's this KAFKA-1382 causing the issue.
Thanks,
Balaji
From: Gulia, Vikram
Sent: Wednesday, September 24, 2014 8:43 AM
To: Sharma, Navdeep; #IT-MAD DES; #IT-MAA
Cc: Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Adding full MAA distro.
DES Offshore looked in to the logs on kafka servers and seems like the issue we encountered yesterday may be described in these threads, please have a look -
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the fix/patch which is available in 0.8.1.2/0.8.2)
Thank You,
Vikram Gulia
From: Sharma, Navdeep
Sent: Wednesday, September 24, 2014 6:53 AM
To: Gulia, Vikram; #IT-MAD DES
Cc: #IT-MAA Offshore; Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Hi Vikram,
We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found that the error occurred only on 23 Sept. This is not historical as we checked last 4 days logs.
It looks like that consumer got stopped on September 22 2014 for Linux patching activty.MAA started consumer September 23 2014 at 1:00 AM.
Issue in server log " BadVersion for /brokers/topics/rain-burn-in/partitions/121/state" but it is not present in previous 4 days logs.
More detail of this error can be found at-
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
We are not sure about data loss in this scenario and working on this.
[cid:image001.png@01CFD7D3.80A9B2B0]
[cid:image002.png@01CFD7D3.80A9B2B0]
Let us know if any concerns.
[cid:image003.gif@01CFD7D3.80A9B2B0]
Navdeep Sharma
Developer - offshore, Middleware Applications & Development
o India: 0120-4532000 - 2234
c: +91-9911698102
From: Gulia, Vikram
Sent: Tuesday, September 23, 2014 6:17 PM
To: #IT-MAD DES
Subject: FW: 9/23 prod issue - offline kafka partitions.
DES Offshore dev,
Please work with MAA offshore to monitor the kafka broker as we had this incident where lot of partitions went offline around 1.45 PM MST and MAA has to restart the kafka servers. We may have lost messages and we need to see if there is a way to figure out what was the impact.
Also, check the logs for kafka servers and see if we can figure out why did partitions go offline or are un-available? Let us know if you find anything relevant.
Thank You,
Vikram Gulia
From: Nielsen, Andy
Sent: Tuesday, September 23, 2014 5:04 PM
To: #IT-MAD DES; Gulia, Vikram
Cc: #IT-MAA
Subject: 9/23 prod issue - offline kafka partitions.
desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
topic: account-access partition: 21 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 51 leader: -1 replicas: 4,6,1 isr:
topic: account-access partition: 81 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 111 leader: -1 replicas: 4,6,1 isr: 1
topic: account-activated partition: 13 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 43 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 73 leader: -1 replicas: 4,6,1 isr:
topic: account-activated partition: 103 leader: -1 replicas: 4,6,1 isr: 1
topic: account-adjustment-issued partition: 27 leader: -1 replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 87 leader: -1 replicas: 4,6,1 isr: 1
topic: account-adjustment-issued partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 11 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 41 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 71 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 101 leader: -1 replicas: 4,6,1 isr:
topic: account-info-updated partition: 7 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 37 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 67 leader: -1 replicas: 4,6,1 isr:
topic: account-info-updated partition: 97 leader: -1 replicas: 4,6,1 isr: 1
topic: account-info-updated partition: 127 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 21 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 51 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 81 leader: -1 replicas: 4,6,1 isr: 1
topic: application-access partition: 111 leader: -1 replicas: 4,6,1 isr: 1
topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 29 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 23 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 53 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 83 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 113 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 32 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 122 leader: -1 replicas: 4,6,1 isr: 1
topic: leads-service partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 84 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1 isr: 1
topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 21 leader: -1 replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 51 leader: -1 replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 81 leader: -1 replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 111 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 25 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 55 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 85 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 115 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 8 leader: -1 replicas: 4,6,1 isr: 1
topic: order-modified partition: 38 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 98 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 128 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 27 leader: -1 replicas: 4,6,1 isr: 1
topic: order-response partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 87 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: outbound-call-attempted partition: 13 leader: -1 replicas: 4,6,1 isr:
topic: outbound-call-attempted partition: 43 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-call-attempted partition: 73 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-call-attempted partition: 103 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 4 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 34 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 64 leader: -1 replicas: 4,6,1 isr:
topic: outbound-communications partition: 94 leader: -1 replicas: 4,6,1 isr: 1
topic: outbound-communications partition: 124 leader: -1 replicas: 4,6,1 isr: 1
topic: postal-mail-undeliverable partition: 15 leader: -1 replicas: 4,6,1 isr: 1
topic: postal-mail-undeliverable partition: 45 leader: -1 replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 75 leader: -1 replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 105 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1 isr: 1
topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 14 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 44 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 74 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 104 leader: -1 replicas: 4,6,1 isr: 1
Andy Nielsen
Middleware Application Admin
303-723-2347
cell:720-971-2856
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
We don’t have GC problem.
zookeeper runs at 0% GC
Kafka broker runs at 1-3% GC
We don’t see many major GC's in our monitoring. We use Concurrent Mark and Sweep GC.
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Tuesday, September 30, 2014 6:41 PM
To: users@kafka.apache.org
Subject: Re: BadVersion state in Kafka Logs
Also check for really long/bad GC pauses as another possibility. Not sure your JDK and JVM_OPTS and if you are setting like this https://kafka.apache.org/documentation.html#java or not. You need to find some "spike" somewhere right before that error happens to track down what is causing the timeouts.
On Tue, Sep 30, 2014 at 6:33 PM, Joe Stein <jo...@stealth.ly> wrote:
> It sounds like you have a much deeper rooted problem. Is zookeeper
> swapping? Something has to be causing this. After you fix this
> symptom you will probably start to see constant leader elections and
> the isr shrinking/growing and constant consumer rebalancing (or at
> least every
> minute) and a herd affect up/down stream occuring. You need to figure
> out what is causing the long session timeout and resolve that, IMHO.
> Zookeeper health is the first place to look. Next would be the network.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Sep 30, 2014 at 4:57 PM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com> wrote:
>
>> The zookeeper session timeout is 60 secs ,but that did not help.
>>
>> We are having broker crash and unresponsive, we got the "conditional
>> update" failed error when broker crashed which confirmed that it is
>> because of KAFKA-1382.
>>
>> server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ -
>> Conditional update of path
>> /brokers/topics/dish-promo-application-access/partitions/128/state
>> with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
>> "leader_epoch":99, "version":1 } and expected version 150 failed due
>> to
>> org.apache.zookeeper.KeeperException$BadVersionException:
>> KeeperErrorCode = BadVersion for
>> /brokers/topics/dish-promo-application-access/partitions/128/state
>>
>> We are in very old version 0.8-beta so it's not just patch but
>> switching to stable release version which also has the patch.
>>
>> -----Original Message-----
>> From: Joe Stein [mailto:joe.stein@stealth.ly]
>> Sent: Tuesday, September 30, 2014 2:01 PM
>> To: users@kafka.apache.org
>> Cc: Neha Narkhede
>> Subject: Re: BadVersion state in Kafka Logs
>>
>> Have you tried increasing your broker's zookeeper session timeout as
>> a work around for now to alleviate the issue? Is that an option for you?
>> Assuming that is the culprit you are timing zk sessions out and
>> bumping into
>> KAFKA-1382 on the reconnect? Not knowing enough about what is going
>> on with the cluster it is hard to say if anything negative will come
>> from it but seems like it might be a an approach to try... if you can
>> figure out what is causing the session to timeout and fix *that* it
>> would be a solution also.... if it is happening every couple days (as
>> another email thread
>> states) something is going on that may not just be fixed by a single
>> patch.
>>
>> /*******************************************
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> ********************************************/
>>
>> On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji <
>> Balaji.Seshadri@dish.com
>> > wrote:
>>
>> > Hi Joe,
>> >
>> > I did not try on 0.8.1 branch ,I can try and see if it goes through
>> > when I get some breather.
>> >
>> > Thanks for initiating on 0.8.1.2.
>> >
>> > Thanks,
>> >
>> > Balaji
>> >
>> > -----Original Message-----
>> > From: Joe Stein [mailto:joe.stein@stealth.ly]
>> > Sent: Tuesday, September 30, 2014 9:34 AM
>> > To: users@kafka.apache.org
>> > Cc: Neha Narkhede
>> > Subject: Re: BadVersion state in Kafka Logs
>> >
>> > Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if
>> > you could make a patch that does would be great.
>> >
>> > I will kick off a discussion for KAFKA-1382 and the scala 2.11 for
>> > 0.8.1.2 release (and see what others may think we should do like
>> > the gradle changes I think we should do too for src release issues
>> > (and the jars in the repo)). I will send that on dev/user in a
>> > little bit (please comment +1 community support please on that
>> > thread for the
>> release).
>> >
>> > /*******************************************
>> > Joe Stein
>> > Founder, Principal Consultant
>> > Big Data Open Source Security LLC
>> > http://www.stealth.ly
>> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> > ********************************************/
>> >
>> > On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <
>> > Balaji.Seshadri@dish.com
>> > > wrote:
>> >
>> > > I would love to help you guys to make Kafka best in Pub/Sub, will
>> > > continue doing that whenever I can.
>> > >
>> > > Do we have 0.8.1.2 release tag or should we apply patch on top
>> > > of
>> > > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
>> > >
>> > > Balaji
>> > >
>> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>> > > Sent: Monday, September 29, 2014 5:21 PM
>> > > To: Seshadri, Balaji
>> > > Cc: users@kafka.apache.org
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > It is difficult to predict an exact date. Though all the
>> > > discussions of the progress and ETA are on the mailing list. You
>> > > can follow the discussions to know the details and/or offer to
>> > > help out on the outstanding issues.
>> > >
>> > > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
>> > > Neha,
>> > >
>> > > Do you know the date in Oct when 0.8.2 is going to be out ?.
>> > >
>> > > Thanks,
>> > >
>> > > Balaji
>> > >
>> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
>> > > neha.narkhede@gmail.com>]
>> > > Sent: Thursday, September 25, 2014 1:08 PM
>> > > To: Seshadri, Balaji
>> > > Cc: users@kafka.apache.org<ma...@kafka.apache.org>
>> > >
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > We are close to the release. I'd probably expect 0.8.2 sometime
>> > > in
>> > October.
>> > >
>> > > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
>> > > Hi Neha,
>> > >
>> > > Do you know when are you guys releasing 0.8.2 ?.
>> > >
>> > > Thanks,
>> > >
>> > > Balaji
>> > >
>> > > -----Original Message-----
>> > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
>> > > Balaji.Seshadri@dish.com>]
>> > > Sent: Thursday, September 25, 2014 9:41 AM
>> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
>> > > Subject: RE: BadVersion state in Kafka Logs
>> > >
>> > > Thanks for the replay.
>> > >
>> > > Please let me know if we can use trunk as 0.8.2 is not yet released.
>> > >
>> > > Balaji
>> > > ________________________________________
>> > > From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
>> > > neha.narkhede@gmail.com>]
>> > > Sent: Wednesday, September 24, 2014 6:32 PM
>> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > From the logs you've attached, my guess is it's most likely due
>> > > to KAFKA-1382.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>
>> > > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > >
>> > > >
>> > > > We got the below error in our logs and our consumers stopped
>> > > > consuming any data ?.It worked only after restart.
>> > > >
>> > > >
>> > > >
>> > > > We would like to confirm that it's because we are running with
>> > > > 0.8-beta version and not 0.8 release version to convince "THE MGMT"
>> > guys.
>> > > >
>> > > >
>> > > >
>> > > > Please let me know if it's this KAFKA-1382 causing the issue.
>> > > >
>> > > >
>> > > >
>> > > > Thanks,
>> > > >
>> > > >
>> > > >
>> > > > Balaji
>> > > >
>> > > >
>> > > >
>> > > > *From:* Gulia, Vikram
>> > > > *Sent:* Wednesday, September 24, 2014 8:43 AM
>> > > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
>> > > > *Cc:* Alam, Mohammad Shah
>> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > Adding full MAA distro.
>> > > >
>> > > >
>> > > >
>> > > > DES Offshore looked in to the logs on kafka servers and seems
>> > > > like the issue we encountered yesterday may be described in
>> > > > these threads, please have a look -
>> > > >
>> > > >
>> > > >
>> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>> > > >
>> > > >
>> > > >
>> > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes
>> > > > the fix/patch which is available in
>> > > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
>> > > >
>> > > >
>> > > >
>> > > > Thank You,
>> > > >
>> > > > Vikram Gulia
>> > > >
>> > > >
>> > > >
>> > > > *From:* Sharma, Navdeep
>> > > > *Sent:* Wednesday, September 24, 2014 6:53 AM
>> > > > *To:* Gulia, Vikram; #IT-MAD DES
>> > > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
>> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > Hi Vikram,
>> > > >
>> > > >
>> > > >
>> > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek)
>> > > > and found that the error occurred only on 23 Sept. This is not
>> > > > historical as we checked last 4 days logs.
>> > > >
>> > > >
>> > > >
>> > > > It looks like that consumer got stopped on September 22 2014
>> > > > for Linux patching activty.MAA started consumer September 23
>> > > > 2014 at
>> > > > 1:00
>> > AM.
>> > > >
>> > > >
>> > > >
>> > > > *Issue *in server log *"* *BadVersion for
>> > > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is
>> > > > not present in previous 4 days logs.
>> > > >
>> > > > More detail of this error can be found at-
>> > > >
>> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>> > > >
>> > > >
>> > > >
>> > > > We are not sure about data loss in this scenario and working on
>> this.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Let us know if any concerns.
>> > > >
>> > > >
>> > > >
>> > > > [image: cid:image001.gif@01CF7B0A.03F21580]
>> > > >
>> > > > Navdeep Sharma
>> > > > Developer - offshore, Middleware Applications & Development o
>> India:
>> > > > 0120-4532000 - 2234
>> > > > c: +91-9911698102<tel:%2B91-9911698102>
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *From:* Gulia, Vikram
>> > > > *Sent:* Tuesday, September 23, 2014 6:17 PM
>> > > > *To:* #IT-MAD DES
>> > > > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > DES Offshore dev,
>> > > >
>> > > >
>> > > >
>> > > > Please work with MAA offshore to monitor the kafka broker as we
>> > > > had this incident where lot of partitions went offline around
>> > > > 1.45 PM MST and MAA has to restart the kafka servers. We may
>> > > > have lost messages and we need to see if there is a way to
>> > > > figure out what was
>> > the impact.
>> > > >
>> > > >
>> > > >
>> > > > Also, check the logs for kafka servers and see if we can figure
>> > > > out why did partitions go offline or are un-available? Let us
>> > > > know if you find anything relevant.
>> > > >
>> > > >
>> > > >
>> > > > Thank You,
>> > > >
>> > > > Vikram Gulia
>> > > >
>> > > >
>> > > >
>> > > > *From:* Nielsen, Andy
>> > > > *Sent:* Tuesday, September 23, 2014 5:04 PM
>> > > > *To:* #IT-MAD DES; Gulia, Vikram
>> > > > *Cc:* #IT-MAA
>> > > > *Subject:* 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
>> > > > --unavailable-partitions
>> > > >
>> > > > topic: account-access partition: 21 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-access partition: 51 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-access partition: 81 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-access partition: 111 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-activated partition: 13 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 43 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 73 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 103 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-adjustment-issued partition: 27 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-adjustment-issued partition: 57 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-adjustment-issued partition: 87 leader: -1
>> > > > replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: account-adjustment-issued partition: 117 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-created partition: 11 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 41 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 71 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 101 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-info-updated partition: 7 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 37 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 67 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-info-updated partition: 97 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 127 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 21 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 51 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 81 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 111 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: bill-generated partition: 3 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 33 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 63 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 93 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 123 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: collected-event partition: 29 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: collected-event partition: 59 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: collected-event partition: 89 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: collected-event partition: 119 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: customer-cues partition: 27 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: customer-cues partition: 57 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: customer-cues partition: 87 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: customer-cues partition: 117 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 23 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 53 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 83 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 113 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: event-response partition: 2 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 32 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 62 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 92 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 122 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: leads-service partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 3 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 33 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: logprod_v3 partition: 63 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 93 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 123 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: online-account-registration-attempted partition: 21
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: online-account-registration-attempted partition: 51
>> leader:
>> > > > -1 replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: online-account-registration-attempted partition: 81
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: online-account-registration-attempted partition: 111
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: order-cancelled partition: 29 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 59 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 89 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 119 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-completed partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-completed partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-completed partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-completed partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 25 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 55 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 85 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 115 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 8 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-modified partition: 38 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 68 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 98 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 128 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-request partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-request partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-request partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-request partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 27 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-response partition: 57 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 87 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 117 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: outbound-call-attempted partition: 13 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-call-attempted partition: 43 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-call-attempted partition: 73 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-call-attempted partition: 103 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 4 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 34 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 64 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 94 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-communications partition: 124 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 15 leader: -1
>> > > > replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 45 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 75 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 105 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: rain-burn-in partition: 4 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-burn-in partition: 34 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-burn-in partition: 64 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-burn-in partition: 94 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-burn-in partition: 124 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-enhanced partition: 26 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-enhanced partition: 56 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-enhanced partition: 86 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-enhanced partition: 116 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-listener partition: 23 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-listener partition: 53 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-listener partition: 83 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-listener partition: 113 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 8 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-load-test partition: 38 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 68 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-load-test partition: 98 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 128 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 2 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 32 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: submit-agreement partition: 62 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 92 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 122 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 14 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 44 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 74 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 104 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > >
>> > > >
>> > > > *Andy Nielsen*
>> > > >
>> > > > *Middleware Application Admin*
>> > > >
>> > > > *303-723-2347<tel:303-723-2347>
>> > > > <303-723-2347<tel:303-723-2347>>*
>> > > >
>> > > > *cell:720-971-2856<tel:720-971-2856>
>> > > > <720-971-2856<tel:720-971-2856>>*
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: BadVersion state in Kafka Logs
Posted by Joe Stein <jo...@stealth.ly>.
Also check for really long/bad GC pauses as another possibility. Not sure
your JDK and JVM_OPTS and if you are setting like this
https://kafka.apache.org/documentation.html#java or not. You need to find
some "spike" somewhere right before that error happens to track down what
is causing the timeouts.
On Tue, Sep 30, 2014 at 6:33 PM, Joe Stein <jo...@stealth.ly> wrote:
> It sounds like you have a much deeper rooted problem. Is zookeeper
> swapping? Something has to be causing this. After you fix this symptom
> you will probably start to see constant leader elections and the isr
> shrinking/growing and constant consumer rebalancing (or at least every
> minute) and a herd affect up/down stream occuring. You need to figure out
> what is causing the long session timeout and resolve that, IMHO. Zookeeper
> health is the first place to look. Next would be the network.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Sep 30, 2014 at 4:57 PM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com> wrote:
>
>> The zookeeper session timeout is 60 secs ,but that did not help.
>>
>> We are having broker crash and unresponsive, we got the "conditional
>> update" failed error when broker crashed which confirmed that it is because
>> of KAFKA-1382.
>>
>> server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ -
>> Conditional update of path
>> /brokers/topics/dish-promo-application-access/partitions/128/state with
>> data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
>> "leader_epoch":99, "version":1 } and expected version 150 failed due to
>> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
>> BadVersion for
>> /brokers/topics/dish-promo-application-access/partitions/128/state
>>
>> We are in very old version 0.8-beta so it's not just patch but switching
>> to stable release version which also has the patch.
>>
>> -----Original Message-----
>> From: Joe Stein [mailto:joe.stein@stealth.ly]
>> Sent: Tuesday, September 30, 2014 2:01 PM
>> To: users@kafka.apache.org
>> Cc: Neha Narkhede
>> Subject: Re: BadVersion state in Kafka Logs
>>
>> Have you tried increasing your broker's zookeeper session timeout as a
>> work around for now to alleviate the issue? Is that an option for you?
>> Assuming that is the culprit you are timing zk sessions out and bumping into
>> KAFKA-1382 on the reconnect? Not knowing enough about what is going on
>> with the cluster it is hard to say if anything negative will come from it
>> but seems like it might be a an approach to try... if you can figure out
>> what is causing the session to timeout and fix *that* it would be a
>> solution also.... if it is happening every couple days (as another email
>> thread
>> states) something is going on that may not just be fixed by a single
>> patch.
>>
>> /*******************************************
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> ********************************************/
>>
>> On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji <
>> Balaji.Seshadri@dish.com
>> > wrote:
>>
>> > Hi Joe,
>> >
>> > I did not try on 0.8.1 branch ,I can try and see if it goes through
>> > when I get some breather.
>> >
>> > Thanks for initiating on 0.8.1.2.
>> >
>> > Thanks,
>> >
>> > Balaji
>> >
>> > -----Original Message-----
>> > From: Joe Stein [mailto:joe.stein@stealth.ly]
>> > Sent: Tuesday, September 30, 2014 9:34 AM
>> > To: users@kafka.apache.org
>> > Cc: Neha Narkhede
>> > Subject: Re: BadVersion state in Kafka Logs
>> >
>> > Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you
>> > could make a patch that does would be great.
>> >
>> > I will kick off a discussion for KAFKA-1382 and the scala 2.11 for
>> > 0.8.1.2 release (and see what others may think we should do like the
>> > gradle changes I think we should do too for src release issues (and
>> > the jars in the repo)). I will send that on dev/user in a little bit
>> > (please comment +1 community support please on that thread for the
>> release).
>> >
>> > /*******************************************
>> > Joe Stein
>> > Founder, Principal Consultant
>> > Big Data Open Source Security LLC
>> > http://www.stealth.ly
>> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> > ********************************************/
>> >
>> > On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <
>> > Balaji.Seshadri@dish.com
>> > > wrote:
>> >
>> > > I would love to help you guys to make Kafka best in Pub/Sub, will
>> > > continue doing that whenever I can.
>> > >
>> > > Do we have 0.8.1.2 release tag or should we apply patch on top of
>> > > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
>> > >
>> > > Balaji
>> > >
>> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>> > > Sent: Monday, September 29, 2014 5:21 PM
>> > > To: Seshadri, Balaji
>> > > Cc: users@kafka.apache.org
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > It is difficult to predict an exact date. Though all the discussions
>> > > of the progress and ETA are on the mailing list. You can follow the
>> > > discussions to know the details and/or offer to help out on the
>> > > outstanding issues.
>> > >
>> > > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
>> > > Neha,
>> > >
>> > > Do you know the date in Oct when 0.8.2 is going to be out ?.
>> > >
>> > > Thanks,
>> > >
>> > > Balaji
>> > >
>> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
>> > > neha.narkhede@gmail.com>]
>> > > Sent: Thursday, September 25, 2014 1:08 PM
>> > > To: Seshadri, Balaji
>> > > Cc: users@kafka.apache.org<ma...@kafka.apache.org>
>> > >
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > We are close to the release. I'd probably expect 0.8.2 sometime in
>> > October.
>> > >
>> > > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
>> > > Hi Neha,
>> > >
>> > > Do you know when are you guys releasing 0.8.2 ?.
>> > >
>> > > Thanks,
>> > >
>> > > Balaji
>> > >
>> > > -----Original Message-----
>> > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
>> > > Balaji.Seshadri@dish.com>]
>> > > Sent: Thursday, September 25, 2014 9:41 AM
>> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
>> > > Subject: RE: BadVersion state in Kafka Logs
>> > >
>> > > Thanks for the replay.
>> > >
>> > > Please let me know if we can use trunk as 0.8.2 is not yet released.
>> > >
>> > > Balaji
>> > > ________________________________________
>> > > From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
>> > > neha.narkhede@gmail.com>]
>> > > Sent: Wednesday, September 24, 2014 6:32 PM
>> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
>> > > Subject: Re: BadVersion state in Kafka Logs
>> > >
>> > > From the logs you've attached, my guess is it's most likely due to
>> > > KAFKA-1382.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
>> > > Balaji.Seshadri@dish.com<ma...@dish.com>
>> > > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > >
>> > > >
>> > > > We got the below error in our logs and our consumers stopped
>> > > > consuming any data ?.It worked only after restart.
>> > > >
>> > > >
>> > > >
>> > > > We would like to confirm that it's because we are running with
>> > > > 0.8-beta version and not 0.8 release version to convince "THE MGMT"
>> > guys.
>> > > >
>> > > >
>> > > >
>> > > > Please let me know if it's this KAFKA-1382 causing the issue.
>> > > >
>> > > >
>> > > >
>> > > > Thanks,
>> > > >
>> > > >
>> > > >
>> > > > Balaji
>> > > >
>> > > >
>> > > >
>> > > > *From:* Gulia, Vikram
>> > > > *Sent:* Wednesday, September 24, 2014 8:43 AM
>> > > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
>> > > > *Cc:* Alam, Mohammad Shah
>> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > Adding full MAA distro.
>> > > >
>> > > >
>> > > >
>> > > > DES Offshore looked in to the logs on kafka servers and seems like
>> > > > the issue we encountered yesterday may be described in these
>> > > > threads, please have a look -
>> > > >
>> > > >
>> > > >
>> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>> > > >
>> > > >
>> > > >
>> > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
>> > > > fix/patch which is available in
>> > > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
>> > > >
>> > > >
>> > > >
>> > > > Thank You,
>> > > >
>> > > > Vikram Gulia
>> > > >
>> > > >
>> > > >
>> > > > *From:* Sharma, Navdeep
>> > > > *Sent:* Wednesday, September 24, 2014 6:53 AM
>> > > > *To:* Gulia, Vikram; #IT-MAD DES
>> > > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
>> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > Hi Vikram,
>> > > >
>> > > >
>> > > >
>> > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek)
>> > > > and found that the error occurred only on 23 Sept. This is not
>> > > > historical as we checked last 4 days logs.
>> > > >
>> > > >
>> > > >
>> > > > It looks like that consumer got stopped on September 22 2014 for
>> > > > Linux patching activty.MAA started consumer September 23 2014 at
>> > > > 1:00
>> > AM.
>> > > >
>> > > >
>> > > >
>> > > > *Issue *in server log *"* *BadVersion for
>> > > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
>> > > > present in previous 4 days logs.
>> > > >
>> > > > More detail of this error can be found at-
>> > > >
>> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>> > > >
>> > > >
>> > > >
>> > > > We are not sure about data loss in this scenario and working on
>> this.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Let us know if any concerns.
>> > > >
>> > > >
>> > > >
>> > > > [image: cid:image001.gif@01CF7B0A.03F21580]
>> > > >
>> > > > Navdeep Sharma
>> > > > Developer - offshore, Middleware Applications & Development o
>> India:
>> > > > 0120-4532000 - 2234
>> > > > c: +91-9911698102<tel:%2B91-9911698102>
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *From:* Gulia, Vikram
>> > > > *Sent:* Tuesday, September 23, 2014 6:17 PM
>> > > > *To:* #IT-MAD DES
>> > > > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > DES Offshore dev,
>> > > >
>> > > >
>> > > >
>> > > > Please work with MAA offshore to monitor the kafka broker as we
>> > > > had this incident where lot of partitions went offline around 1.45
>> > > > PM MST and MAA has to restart the kafka servers. We may have lost
>> > > > messages and we need to see if there is a way to figure out what
>> > > > was
>> > the impact.
>> > > >
>> > > >
>> > > >
>> > > > Also, check the logs for kafka servers and see if we can figure
>> > > > out why did partitions go offline or are un-available? Let us know
>> > > > if you find anything relevant.
>> > > >
>> > > >
>> > > >
>> > > > Thank You,
>> > > >
>> > > > Vikram Gulia
>> > > >
>> > > >
>> > > >
>> > > > *From:* Nielsen, Andy
>> > > > *Sent:* Tuesday, September 23, 2014 5:04 PM
>> > > > *To:* #IT-MAD DES; Gulia, Vikram
>> > > > *Cc:* #IT-MAA
>> > > > *Subject:* 9/23 prod issue - offline kafka partitions.
>> > > >
>> > > >
>> > > >
>> > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
>> > > > --unavailable-partitions
>> > > >
>> > > > topic: account-access partition: 21 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-access partition: 51 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-access partition: 81 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-access partition: 111 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: account-activated partition: 13 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 43 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 73 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-activated partition: 103 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-adjustment-issued partition: 27 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-adjustment-issued partition: 57 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-adjustment-issued partition: 87 leader: -1
>> > > > replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: account-adjustment-issued partition: 117 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: account-created partition: 11 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 41 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 71 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-created partition: 101 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: account-info-updated partition: 7 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 37 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 67 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: account-info-updated partition: 97 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: account-info-updated partition: 127 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 21 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 51 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 81 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: application-access partition: 111 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: bill-generated partition: 3 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 33 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 63 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 93 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: bill-generated partition: 123 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: collected-event partition: 29 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: collected-event partition: 59 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: collected-event partition: 89 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: collected-event partition: 119 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: customer-cues partition: 27 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: customer-cues partition: 57 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: customer-cues partition: 87 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: customer-cues partition: 117 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 23 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 53 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 83 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: dish-promo-application-access partition: 113 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: event-response partition: 2 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 32 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 62 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 92 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: event-response partition: 122 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: leads-service partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: leads-service partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 3 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 33 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: logprod_v3 partition: 63 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 93 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: logprod_v3 partition: 123 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: online-account-registration-attempted partition: 21
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: online-account-registration-attempted partition: 51
>> leader:
>> > > > -1 replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: online-account-registration-attempted partition: 81
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: online-account-registration-attempted partition: 111
>> leader:
>> > > > -1 replicas: 4,6,1 isr:
>> > > >
>> > > > topic: order-cancelled partition: 29 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 59 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 89 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-cancelled partition: 119 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-completed partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-completed partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-completed partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-completed partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 25 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 55 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 85 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-created partition: 115 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 8 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-modified partition: 38 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 68 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 98 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-modified partition: 128 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-request partition: 24 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-request partition: 54 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-request partition: 84 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-request partition: 114 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 27 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: order-response partition: 57 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 87 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: order-response partition: 117 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: outbound-call-attempted partition: 13 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-call-attempted partition: 43 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-call-attempted partition: 73 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-call-attempted partition: 103 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 4 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 34 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 64 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: outbound-communications partition: 94 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: outbound-communications partition: 124 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 15 leader: -1
>> > > > replicas: 4,6,1 isr: 1
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 45 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 75 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: postal-mail-undeliverable partition: 105 leader: -1
>> > > > replicas: 4,6,1 isr:
>> > > >
>> > > > topic: rain-burn-in partition: 4 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-burn-in partition: 34 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-burn-in partition: 64 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-burn-in partition: 94 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-burn-in partition: 124 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-enhanced partition: 26 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-enhanced partition: 56 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-enhanced partition: 86 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-enhanced partition: 116 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-listener partition: 23 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-listener partition: 53 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-listener partition: 83 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-listener partition: 113 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 8 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-load-test partition: 38 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 68 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: rain-load-test partition: 98 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: rain-load-test partition: 128 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 2 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 32 leader: -1 replicas:
>> 4,6,1
>> > > > isr: 1
>> > > >
>> > > > topic: submit-agreement partition: 62 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 92 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: submit-agreement partition: 122 leader: -1 replicas:
>> 4,6,1
>> > > > isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 14 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 44 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 74 leader: -1
>> > replicas:
>> > > > 4,6,1 isr:
>> > > >
>> > > > topic: threshold-exceeded partition: 104 leader: -1
>> > replicas:
>> > > > 4,6,1 isr: 1
>> > > >
>> > > >
>> > > >
>> > > > *Andy Nielsen*
>> > > >
>> > > > *Middleware Application Admin*
>> > > >
>> > > > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
>> > > >
>> > > > *cell:720-971-2856<tel:720-971-2856>
>> > > > <720-971-2856<tel:720-971-2856>>*
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: BadVersion state in Kafka Logs
Posted by Joe Stein <jo...@stealth.ly>.
It sounds like you have a much deeper rooted problem. Is zookeeper
swapping? Something has to be causing this. After you fix this symptom
you will probably start to see constant leader elections and the isr
shrinking/growing and constant consumer rebalancing (or at least every
minute) and a herd affect up/down stream occuring. You need to figure out
what is causing the long session timeout and resolve that, IMHO. Zookeeper
health is the first place to look. Next would be the network.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Sep 30, 2014 at 4:57 PM, Seshadri, Balaji <Ba...@dish.com>
wrote:
> The zookeeper session timeout is 60 secs ,but that did not help.
>
> We are having broker crash and unresponsive, we got the "conditional
> update" failed error when broker crashed which confirmed that it is because
> of KAFKA-1382.
>
> server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ -
> Conditional update of path
> /brokers/topics/dish-promo-application-access/partitions/128/state with
> data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
> "leader_epoch":99, "version":1 } and expected version 150 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for
> /brokers/topics/dish-promo-application-access/partitions/128/state
>
> We are in very old version 0.8-beta so it's not just patch but switching
> to stable release version which also has the patch.
>
> -----Original Message-----
> From: Joe Stein [mailto:joe.stein@stealth.ly]
> Sent: Tuesday, September 30, 2014 2:01 PM
> To: users@kafka.apache.org
> Cc: Neha Narkhede
> Subject: Re: BadVersion state in Kafka Logs
>
> Have you tried increasing your broker's zookeeper session timeout as a
> work around for now to alleviate the issue? Is that an option for you?
> Assuming that is the culprit you are timing zk sessions out and bumping into
> KAFKA-1382 on the reconnect? Not knowing enough about what is going on
> with the cluster it is hard to say if anything negative will come from it
> but seems like it might be a an approach to try... if you can figure out
> what is causing the session to timeout and fix *that* it would be a
> solution also.... if it is happening every couple days (as another email
> thread
> states) something is going on that may not just be fixed by a single patch.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > Hi Joe,
> >
> > I did not try on 0.8.1 branch ,I can try and see if it goes through
> > when I get some breather.
> >
> > Thanks for initiating on 0.8.1.2.
> >
> > Thanks,
> >
> > Balaji
> >
> > -----Original Message-----
> > From: Joe Stein [mailto:joe.stein@stealth.ly]
> > Sent: Tuesday, September 30, 2014 9:34 AM
> > To: users@kafka.apache.org
> > Cc: Neha Narkhede
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you
> > could make a patch that does would be great.
> >
> > I will kick off a discussion for KAFKA-1382 and the scala 2.11 for
> > 0.8.1.2 release (and see what others may think we should do like the
> > gradle changes I think we should do too for src release issues (and
> > the jars in the repo)). I will send that on dev/user in a little bit
> > (please comment +1 community support please on that thread for the
> release).
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> > On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com
> > > wrote:
> >
> > > I would love to help you guys to make Kafka best in Pub/Sub, will
> > > continue doing that whenever I can.
> > >
> > > Do we have 0.8.1.2 release tag or should we apply patch on top of
> > > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
> > >
> > > Balaji
> > >
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Monday, September 29, 2014 5:21 PM
> > > To: Seshadri, Balaji
> > > Cc: users@kafka.apache.org
> > > Subject: Re: BadVersion state in Kafka Logs
> > >
> > > It is difficult to predict an exact date. Though all the discussions
> > > of the progress and ETA are on the mailing list. You can follow the
> > > discussions to know the details and/or offer to help out on the
> > > outstanding issues.
> > >
> > > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > > Neha,
> > >
> > > Do you know the date in Oct when 0.8.2 is going to be out ?.
> > >
> > > Thanks,
> > >
> > > Balaji
> > >
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
> > > neha.narkhede@gmail.com>]
> > > Sent: Thursday, September 25, 2014 1:08 PM
> > > To: Seshadri, Balaji
> > > Cc: users@kafka.apache.org<ma...@kafka.apache.org>
> > >
> > > Subject: Re: BadVersion state in Kafka Logs
> > >
> > > We are close to the release. I'd probably expect 0.8.2 sometime in
> > October.
> > >
> > > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> > > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > > Hi Neha,
> > >
> > > Do you know when are you guys releasing 0.8.2 ?.
> > >
> > > Thanks,
> > >
> > > Balaji
> > >
> > > -----Original Message-----
> > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
> > > Balaji.Seshadri@dish.com>]
> > > Sent: Thursday, September 25, 2014 9:41 AM
> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > > Subject: RE: BadVersion state in Kafka Logs
> > >
> > > Thanks for the replay.
> > >
> > > Please let me know if we can use trunk as 0.8.2 is not yet released.
> > >
> > > Balaji
> > > ________________________________________
> > > From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
> > > neha.narkhede@gmail.com>]
> > > Sent: Wednesday, September 24, 2014 6:32 PM
> > > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > > Subject: Re: BadVersion state in Kafka Logs
> > >
> > > From the logs you've attached, my guess is it's most likely due to
> > > KAFKA-1382.
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> > > Balaji.Seshadri@dish.com<ma...@dish.com>
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > We got the below error in our logs and our consumers stopped
> > > > consuming any data ?.It worked only after restart.
> > > >
> > > >
> > > >
> > > > We would like to confirm that it's because we are running with
> > > > 0.8-beta version and not 0.8 release version to convince "THE MGMT"
> > guys.
> > > >
> > > >
> > > >
> > > > Please let me know if it's this KAFKA-1382 causing the issue.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > Balaji
> > > >
> > > >
> > > >
> > > > *From:* Gulia, Vikram
> > > > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > > > *Cc:* Alam, Mohammad Shah
> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > > >
> > > >
> > > >
> > > > Adding full MAA distro.
> > > >
> > > >
> > > >
> > > > DES Offshore looked in to the logs on kafka servers and seems like
> > > > the issue we encountered yesterday may be described in these
> > > > threads, please have a look -
> > > >
> > > >
> > > >
> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > > >
> > > >
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > > > fix/patch which is available in
> > > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
> > > >
> > > >
> > > >
> > > > Thank You,
> > > >
> > > > Vikram Gulia
> > > >
> > > >
> > > >
> > > > *From:* Sharma, Navdeep
> > > > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > > > *To:* Gulia, Vikram; #IT-MAD DES
> > > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > > >
> > > >
> > > >
> > > > Hi Vikram,
> > > >
> > > >
> > > >
> > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek)
> > > > and found that the error occurred only on 23 Sept. This is not
> > > > historical as we checked last 4 days logs.
> > > >
> > > >
> > > >
> > > > It looks like that consumer got stopped on September 22 2014 for
> > > > Linux patching activty.MAA started consumer September 23 2014 at
> > > > 1:00
> > AM.
> > > >
> > > >
> > > >
> > > > *Issue *in server log *"* *BadVersion for
> > > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > > > present in previous 4 days logs.
> > > >
> > > > More detail of this error can be found at-
> > > >
> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > > >
> > > >
> > > >
> > > > We are not sure about data loss in this scenario and working on this.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Let us know if any concerns.
> > > >
> > > >
> > > >
> > > > [image: cid:image001.gif@01CF7B0A.03F21580]
> > > >
> > > > Navdeep Sharma
> > > > Developer - offshore, Middleware Applications & Development o India:
> > > > 0120-4532000 - 2234
> > > > c: +91-9911698102<tel:%2B91-9911698102>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *From:* Gulia, Vikram
> > > > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > > > *To:* #IT-MAD DES
> > > > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> > > >
> > > >
> > > >
> > > > DES Offshore dev,
> > > >
> > > >
> > > >
> > > > Please work with MAA offshore to monitor the kafka broker as we
> > > > had this incident where lot of partitions went offline around 1.45
> > > > PM MST and MAA has to restart the kafka servers. We may have lost
> > > > messages and we need to see if there is a way to figure out what
> > > > was
> > the impact.
> > > >
> > > >
> > > >
> > > > Also, check the logs for kafka servers and see if we can figure
> > > > out why did partitions go offline or are un-available? Let us know
> > > > if you find anything relevant.
> > > >
> > > >
> > > >
> > > > Thank You,
> > > >
> > > > Vikram Gulia
> > > >
> > > >
> > > >
> > > > *From:* Nielsen, Andy
> > > > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > > > *To:* #IT-MAD DES; Gulia, Vikram
> > > > *Cc:* #IT-MAA
> > > > *Subject:* 9/23 prod issue - offline kafka partitions.
> > > >
> > > >
> > > >
> > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
> > > > --unavailable-partitions
> > > >
> > > > topic: account-access partition: 21 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: account-access partition: 51 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: account-access partition: 81 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: account-access partition: 111 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: account-activated partition: 13 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: account-activated partition: 43 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: account-activated partition: 73 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: account-activated partition: 103 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: account-adjustment-issued partition: 27 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: account-adjustment-issued partition: 57 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: account-adjustment-issued partition: 87 leader: -1
> > > > replicas: 4,6,1 isr: 1
> > > >
> > > > topic: account-adjustment-issued partition: 117 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: account-created partition: 11 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: account-created partition: 41 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: account-created partition: 71 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: account-created partition: 101 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: account-info-updated partition: 7 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: account-info-updated partition: 37 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: account-info-updated partition: 67 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: account-info-updated partition: 97 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: account-info-updated partition: 127 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: application-access partition: 21 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: application-access partition: 51 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: application-access partition: 81 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: application-access partition: 111 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: bill-generated partition: 3 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: bill-generated partition: 33 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: bill-generated partition: 63 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: bill-generated partition: 93 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: bill-generated partition: 123 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: collected-event partition: 29 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: collected-event partition: 59 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: collected-event partition: 89 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: collected-event partition: 119 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: customer-cues partition: 27 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: customer-cues partition: 57 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: customer-cues partition: 87 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: customer-cues partition: 117 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: dish-promo-application-access partition: 23 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: dish-promo-application-access partition: 53 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: dish-promo-application-access partition: 83 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: dish-promo-application-access partition: 113 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: event-response partition: 2 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: event-response partition: 32 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: event-response partition: 62 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: event-response partition: 92 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: event-response partition: 122 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: leads-service partition: 24 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: leads-service partition: 54 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: leads-service partition: 84 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: leads-service partition: 114 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: logprod_v3 partition: 3 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: logprod_v3 partition: 33 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: logprod_v3 partition: 63 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: logprod_v3 partition: 93 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: logprod_v3 partition: 123 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: online-account-registration-attempted partition: 21
> leader:
> > > > -1 replicas: 4,6,1 isr:
> > > >
> > > > topic: online-account-registration-attempted partition: 51
> leader:
> > > > -1 replicas: 4,6,1 isr: 1
> > > >
> > > > topic: online-account-registration-attempted partition: 81
> leader:
> > > > -1 replicas: 4,6,1 isr:
> > > >
> > > > topic: online-account-registration-attempted partition: 111
> leader:
> > > > -1 replicas: 4,6,1 isr:
> > > >
> > > > topic: order-cancelled partition: 29 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-cancelled partition: 59 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-cancelled partition: 89 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-cancelled partition: 119 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-completed partition: 24 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-completed partition: 54 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-completed partition: 84 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-completed partition: 114 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-created partition: 25 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-created partition: 55 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-created partition: 85 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-created partition: 115 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-modified partition: 8 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-modified partition: 38 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-modified partition: 68 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-modified partition: 98 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-modified partition: 128 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-request partition: 24 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-request partition: 54 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-request partition: 84 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-request partition: 114 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-response partition: 27 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: order-response partition: 57 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-response partition: 87 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: order-response partition: 117 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: outbound-call-attempted partition: 13 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: outbound-call-attempted partition: 43 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: outbound-call-attempted partition: 73 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: outbound-call-attempted partition: 103 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: outbound-communications partition: 4 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: outbound-communications partition: 34 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: outbound-communications partition: 64 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: outbound-communications partition: 94 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: outbound-communications partition: 124 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > > topic: postal-mail-undeliverable partition: 15 leader: -1
> > > > replicas: 4,6,1 isr: 1
> > > >
> > > > topic: postal-mail-undeliverable partition: 45 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: postal-mail-undeliverable partition: 75 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: postal-mail-undeliverable partition: 105 leader: -1
> > > > replicas: 4,6,1 isr:
> > > >
> > > > topic: rain-burn-in partition: 4 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-burn-in partition: 34 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-burn-in partition: 64 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-burn-in partition: 94 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-burn-in partition: 124 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-enhanced partition: 26 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-enhanced partition: 56 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-enhanced partition: 86 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-enhanced partition: 116 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-listener partition: 23 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-listener partition: 53 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-listener partition: 83 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-listener partition: 113 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-load-test partition: 8 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-load-test partition: 38 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-load-test partition: 68 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: rain-load-test partition: 98 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: rain-load-test partition: 128 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: submit-agreement partition: 2 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: submit-agreement partition: 32 leader: -1 replicas:
> 4,6,1
> > > > isr: 1
> > > >
> > > > topic: submit-agreement partition: 62 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: submit-agreement partition: 92 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: submit-agreement partition: 122 leader: -1 replicas:
> 4,6,1
> > > > isr:
> > > >
> > > > topic: threshold-exceeded partition: 14 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: threshold-exceeded partition: 44 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: threshold-exceeded partition: 74 leader: -1
> > replicas:
> > > > 4,6,1 isr:
> > > >
> > > > topic: threshold-exceeded partition: 104 leader: -1
> > replicas:
> > > > 4,6,1 isr: 1
> > > >
> > > >
> > > >
> > > > *Andy Nielsen*
> > > >
> > > > *Middleware Application Admin*
> > > >
> > > > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
> > > >
> > > > *cell:720-971-2856<tel:720-971-2856>
> > > > <720-971-2856<tel:720-971-2856>>*
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
The zookeeper session timeout is 60 secs ,but that did not help.
We are having broker crash and unresponsive, we got the "conditional update" failed error when broker crashed which confirmed that it is because of KAFKA-1382.
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional update of path /brokers/topics/dish-promo-application-access/partitions/128/state with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99, "version":1 } and expected version 150 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/dish-promo-application-access/partitions/128/state
We are in very old version 0.8-beta so it's not just patch but switching to stable release version which also has the patch.
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Tuesday, September 30, 2014 2:01 PM
To: users@kafka.apache.org
Cc: Neha Narkhede
Subject: Re: BadVersion state in Kafka Logs
Have you tried increasing your broker's zookeeper session timeout as a work around for now to alleviate the issue? Is that an option for you? Assuming that is the culprit you are timing zk sessions out and bumping into
KAFKA-1382 on the reconnect? Not knowing enough about what is going on with the cluster it is hard to say if anything negative will come from it but seems like it might be a an approach to try... if you can figure out what is causing the session to timeout and fix *that* it would be a solution also.... if it is happening every couple days (as another email thread
states) something is going on that may not just be fixed by a single patch.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi Joe,
>
> I did not try on 0.8.1 branch ,I can try and see if it goes through
> when I get some breather.
>
> Thanks for initiating on 0.8.1.2.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Joe Stein [mailto:joe.stein@stealth.ly]
> Sent: Tuesday, September 30, 2014 9:34 AM
> To: users@kafka.apache.org
> Cc: Neha Narkhede
> Subject: Re: BadVersion state in Kafka Logs
>
> Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you
> could make a patch that does would be great.
>
> I will kick off a discussion for KAFKA-1382 and the scala 2.11 for
> 0.8.1.2 release (and see what others may think we should do like the
> gradle changes I think we should do too for src release issues (and
> the jars in the repo)). I will send that on dev/user in a little bit
> (please comment +1 community support please on that thread for the release).
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > I would love to help you guys to make Kafka best in Pub/Sub, will
> > continue doing that whenever I can.
> >
> > Do we have 0.8.1.2 release tag or should we apply patch on top of
> > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
> >
> > Balaji
> >
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Monday, September 29, 2014 5:21 PM
> > To: Seshadri, Balaji
> > Cc: users@kafka.apache.org
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > It is difficult to predict an exact date. Though all the discussions
> > of the progress and ETA are on the mailing list. You can follow the
> > discussions to know the details and/or offer to help out on the
> > outstanding issues.
> >
> > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > Neha,
> >
> > Do you know the date in Oct when 0.8.2 is going to be out ?.
> >
> > Thanks,
> >
> > Balaji
> >
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
> > neha.narkhede@gmail.com>]
> > Sent: Thursday, September 25, 2014 1:08 PM
> > To: Seshadri, Balaji
> > Cc: users@kafka.apache.org<ma...@kafka.apache.org>
> >
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > We are close to the release. I'd probably expect 0.8.2 sometime in
> October.
> >
> > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > Hi Neha,
> >
> > Do you know when are you guys releasing 0.8.2 ?.
> >
> > Thanks,
> >
> > Balaji
> >
> > -----Original Message-----
> > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
> > Balaji.Seshadri@dish.com>]
> > Sent: Thursday, September 25, 2014 9:41 AM
> > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > Subject: RE: BadVersion state in Kafka Logs
> >
> > Thanks for the replay.
> >
> > Please let me know if we can use trunk as 0.8.2 is not yet released.
> >
> > Balaji
> > ________________________________________
> > From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
> > neha.narkhede@gmail.com>]
> > Sent: Wednesday, September 24, 2014 6:32 PM
> > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > From the logs you've attached, my guess is it's most likely due to
> > KAFKA-1382.
> >
> > Thanks,
> > Neha
> >
> > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>
> > > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > We got the below error in our logs and our consumers stopped
> > > consuming any data ?.It worked only after restart.
> > >
> > >
> > >
> > > We would like to confirm that it's because we are running with
> > > 0.8-beta version and not 0.8 release version to convince "THE MGMT"
> guys.
> > >
> > >
> > >
> > > Please let me know if it's this KAFKA-1382 causing the issue.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Balaji
> > >
> > >
> > >
> > > *From:* Gulia, Vikram
> > > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > > *Cc:* Alam, Mohammad Shah
> > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > Adding full MAA distro.
> > >
> > >
> > >
> > > DES Offshore looked in to the logs on kafka servers and seems like
> > > the issue we encountered yesterday may be described in these
> > > threads, please have a look -
> > >
> > >
> > >
> > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > >
> > >
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > > fix/patch which is available in
> > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
> > >
> > >
> > >
> > > Thank You,
> > >
> > > Vikram Gulia
> > >
> > >
> > >
> > > *From:* Sharma, Navdeep
> > > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > > *To:* Gulia, Vikram; #IT-MAD DES
> > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > Hi Vikram,
> > >
> > >
> > >
> > > We analyzed below mentioned issue with MAA-Offshore (Abhishek)
> > > and found that the error occurred only on 23 Sept. This is not
> > > historical as we checked last 4 days logs.
> > >
> > >
> > >
> > > It looks like that consumer got stopped on September 22 2014 for
> > > Linux patching activty.MAA started consumer September 23 2014 at
> > > 1:00
> AM.
> > >
> > >
> > >
> > > *Issue *in server log *"* *BadVersion for
> > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > > present in previous 4 days logs.
> > >
> > > More detail of this error can be found at-
> > >
> > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > >
> > >
> > >
> > > We are not sure about data loss in this scenario and working on this.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Let us know if any concerns.
> > >
> > >
> > >
> > > [image: cid:image001.gif@01CF7B0A.03F21580]
> > >
> > > Navdeep Sharma
> > > Developer - offshore, Middleware Applications & Development o India:
> > > 0120-4532000 - 2234
> > > c: +91-9911698102<tel:%2B91-9911698102>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *From:* Gulia, Vikram
> > > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > > *To:* #IT-MAD DES
> > > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > DES Offshore dev,
> > >
> > >
> > >
> > > Please work with MAA offshore to monitor the kafka broker as we
> > > had this incident where lot of partitions went offline around 1.45
> > > PM MST and MAA has to restart the kafka servers. We may have lost
> > > messages and we need to see if there is a way to figure out what
> > > was
> the impact.
> > >
> > >
> > >
> > > Also, check the logs for kafka servers and see if we can figure
> > > out why did partitions go offline or are un-available? Let us know
> > > if you find anything relevant.
> > >
> > >
> > >
> > > Thank You,
> > >
> > > Vikram Gulia
> > >
> > >
> > >
> > > *From:* Nielsen, Andy
> > > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > > *To:* #IT-MAD DES; Gulia, Vikram
> > > *Cc:* #IT-MAA
> > > *Subject:* 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
> > > --unavailable-partitions
> > >
> > > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-activated partition: 13 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 43 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 73 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 103 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-adjustment-issued partition: 27 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-adjustment-issued partition: 57 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-adjustment-issued partition: 87 leader: -1
> > > replicas: 4,6,1 isr: 1
> > >
> > > topic: account-adjustment-issued partition: 117 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-info-updated partition: 7 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 37 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 67 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-info-updated partition: 97 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 127 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 21 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 51 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 81 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 111 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: dish-promo-application-access partition: 23 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 53 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 83 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 113 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: online-account-registration-attempted partition: 21 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: online-account-registration-attempted partition: 51 leader:
> > > -1 replicas: 4,6,1 isr: 1
> > >
> > > topic: online-account-registration-attempted partition: 81 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: online-account-registration-attempted partition: 111 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: outbound-call-attempted partition: 13 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-call-attempted partition: 43 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-call-attempted partition: 73 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-call-attempted partition: 103 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 4 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 34 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 64 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 94 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-communications partition: 124 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: postal-mail-undeliverable partition: 15 leader: -1
> > > replicas: 4,6,1 isr: 1
> > >
> > > topic: postal-mail-undeliverable partition: 45 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: postal-mail-undeliverable partition: 75 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: postal-mail-undeliverable partition: 105 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: threshold-exceeded partition: 14 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 44 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 74 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 104 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > >
> > >
> > > *Andy Nielsen*
> > >
> > > *Middleware Application Admin*
> > >
> > > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
> > >
> > > *cell:720-971-2856<tel:720-971-2856>
> > > <720-971-2856<tel:720-971-2856>>*
> > >
> > >
> > >
> >
> >
> >
>
Re: BadVersion state in Kafka Logs
Posted by Joe Stein <jo...@stealth.ly>.
Have you tried increasing your broker's zookeeper session timeout as a work
around for now to alleviate the issue? Is that an option for you? Assuming
that is the culprit you are timing zk sessions out and bumping into
KAFKA-1382 on the reconnect? Not knowing enough about what is going on with
the cluster it is hard to say if anything negative will come from it but
seems like it might be a an approach to try... if you can figure out what
is causing the session to timeout and fix *that* it would be a solution
also.... if it is happening every couple days (as another email thread
states) something is going on that may not just be fixed by a single patch.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi Joe,
>
> I did not try on 0.8.1 branch ,I can try and see if it goes through when I
> get some breather.
>
> Thanks for initiating on 0.8.1.2.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Joe Stein [mailto:joe.stein@stealth.ly]
> Sent: Tuesday, September 30, 2014 9:34 AM
> To: users@kafka.apache.org
> Cc: Neha Narkhede
> Subject: Re: BadVersion state in Kafka Logs
>
> Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you
> could make a patch that does would be great.
>
> I will kick off a discussion for KAFKA-1382 and the scala 2.11 for 0.8.1.2
> release (and see what others may think we should do like the gradle changes
> I think we should do too for src release issues (and the jars in the
> repo)). I will send that on dev/user in a little bit (please comment +1
> community support please on that thread for the release).
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > I would love to help you guys to make Kafka best in Pub/Sub, will
> > continue doing that whenever I can.
> >
> > Do we have 0.8.1.2 release tag or should we apply patch on top of
> > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
> >
> > Balaji
> >
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Monday, September 29, 2014 5:21 PM
> > To: Seshadri, Balaji
> > Cc: users@kafka.apache.org
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > It is difficult to predict an exact date. Though all the discussions
> > of the progress and ETA are on the mailing list. You can follow the
> > discussions to know the details and/or offer to help out on the
> > outstanding issues.
> >
> > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > Neha,
> >
> > Do you know the date in Oct when 0.8.2 is going to be out ?.
> >
> > Thanks,
> >
> > Balaji
> >
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
> > neha.narkhede@gmail.com>]
> > Sent: Thursday, September 25, 2014 1:08 PM
> > To: Seshadri, Balaji
> > Cc: users@kafka.apache.org<ma...@kafka.apache.org>
> >
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > We are close to the release. I'd probably expect 0.8.2 sometime in
> October.
> >
> > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> > Hi Neha,
> >
> > Do you know when are you guys releasing 0.8.2 ?.
> >
> > Thanks,
> >
> > Balaji
> >
> > -----Original Message-----
> > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
> > Balaji.Seshadri@dish.com>]
> > Sent: Thursday, September 25, 2014 9:41 AM
> > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > Subject: RE: BadVersion state in Kafka Logs
> >
> > Thanks for the replay.
> >
> > Please let me know if we can use trunk as 0.8.2 is not yet released.
> >
> > Balaji
> > ________________________________________
> > From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
> > neha.narkhede@gmail.com>]
> > Sent: Wednesday, September 24, 2014 6:32 PM
> > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > Subject: Re: BadVersion state in Kafka Logs
> >
> > From the logs you've attached, my guess is it's most likely due to
> > KAFKA-1382.
> >
> > Thanks,
> > Neha
> >
> > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> > Balaji.Seshadri@dish.com<ma...@dish.com>
> > > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > We got the below error in our logs and our consumers stopped
> > > consuming any data ?.It worked only after restart.
> > >
> > >
> > >
> > > We would like to confirm that it's because we are running with
> > > 0.8-beta version and not 0.8 release version to convince "THE MGMT"
> guys.
> > >
> > >
> > >
> > > Please let me know if it's this KAFKA-1382 causing the issue.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Balaji
> > >
> > >
> > >
> > > *From:* Gulia, Vikram
> > > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > > *Cc:* Alam, Mohammad Shah
> > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > Adding full MAA distro.
> > >
> > >
> > >
> > > DES Offshore looked in to the logs on kafka servers and seems like
> > > the issue we encountered yesterday may be described in these
> > > threads, please have a look -
> > >
> > >
> > >
> > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > >
> > >
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > > fix/patch which is available in 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
> > >
> > >
> > >
> > > Thank You,
> > >
> > > Vikram Gulia
> > >
> > >
> > >
> > > *From:* Sharma, Navdeep
> > > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > > *To:* Gulia, Vikram; #IT-MAD DES
> > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > Hi Vikram,
> > >
> > >
> > >
> > > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> > > found that the error occurred only on 23 Sept. This is not
> > > historical as we checked last 4 days logs.
> > >
> > >
> > >
> > > It looks like that consumer got stopped on September 22 2014 for
> > > Linux patching activty.MAA started consumer September 23 2014 at 1:00
> AM.
> > >
> > >
> > >
> > > *Issue *in server log *"* *BadVersion for
> > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > > present in previous 4 days logs.
> > >
> > > More detail of this error can be found at-
> > >
> > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> > >
> > >
> > >
> > > We are not sure about data loss in this scenario and working on this.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Let us know if any concerns.
> > >
> > >
> > >
> > > [image: cid:image001.gif@01CF7B0A.03F21580]
> > >
> > > Navdeep Sharma
> > > Developer - offshore, Middleware Applications & Development o India:
> > > 0120-4532000 - 2234
> > > c: +91-9911698102<tel:%2B91-9911698102>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *From:* Gulia, Vikram
> > > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > > *To:* #IT-MAD DES
> > > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > DES Offshore dev,
> > >
> > >
> > >
> > > Please work with MAA offshore to monitor the kafka broker as we had
> > > this incident where lot of partitions went offline around 1.45 PM
> > > MST and MAA has to restart the kafka servers. We may have lost
> > > messages and we need to see if there is a way to figure out what was
> the impact.
> > >
> > >
> > >
> > > Also, check the logs for kafka servers and see if we can figure out
> > > why did partitions go offline or are un-available? Let us know if
> > > you find anything relevant.
> > >
> > >
> > >
> > > Thank You,
> > >
> > > Vikram Gulia
> > >
> > >
> > >
> > > *From:* Nielsen, Andy
> > > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > > *To:* #IT-MAD DES; Gulia, Vikram
> > > *Cc:* #IT-MAA
> > > *Subject:* 9/23 prod issue - offline kafka partitions.
> > >
> > >
> > >
> > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
> > > --unavailable-partitions
> > >
> > > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: account-activated partition: 13 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 43 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 73 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-activated partition: 103 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-adjustment-issued partition: 27 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-adjustment-issued partition: 57 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-adjustment-issued partition: 87 leader: -1
> > > replicas: 4,6,1 isr: 1
> > >
> > > topic: account-adjustment-issued partition: 117 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: account-info-updated partition: 7 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 37 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 67 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: account-info-updated partition: 97 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: account-info-updated partition: 127 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 21 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 51 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 81 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: application-access partition: 111 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: dish-promo-application-access partition: 23 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 53 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 83 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: dish-promo-application-access partition: 113 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: online-account-registration-attempted partition: 21 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: online-account-registration-attempted partition: 51 leader:
> > > -1 replicas: 4,6,1 isr: 1
> > >
> > > topic: online-account-registration-attempted partition: 81 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: online-account-registration-attempted partition: 111 leader:
> > > -1 replicas: 4,6,1 isr:
> > >
> > > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: outbound-call-attempted partition: 13 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-call-attempted partition: 43 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-call-attempted partition: 73 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-call-attempted partition: 103 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 4 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 34 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 64 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: outbound-communications partition: 94 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: outbound-communications partition: 124 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > > topic: postal-mail-undeliverable partition: 15 leader: -1
> > > replicas: 4,6,1 isr: 1
> > >
> > > topic: postal-mail-undeliverable partition: 45 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: postal-mail-undeliverable partition: 75 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: postal-mail-undeliverable partition: 105 leader: -1
> > > replicas: 4,6,1 isr:
> > >
> > > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > > isr: 1
> > >
> > > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > > isr:
> > >
> > > topic: threshold-exceeded partition: 14 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 44 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 74 leader: -1
> replicas:
> > > 4,6,1 isr:
> > >
> > > topic: threshold-exceeded partition: 104 leader: -1
> replicas:
> > > 4,6,1 isr: 1
> > >
> > >
> > >
> > > *Andy Nielsen*
> > >
> > > *Middleware Application Admin*
> > >
> > > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
> > >
> > > *cell:720-971-2856<tel:720-971-2856>
> > > <720-971-2856<tel:720-971-2856>>*
> > >
> > >
> > >
> >
> >
> >
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
Hi Joe,
I did not try on 0.8.1 branch ,I can try and see if it goes through when I get some breather.
Thanks for initiating on 0.8.1.2.
Thanks,
Balaji
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Tuesday, September 30, 2014 9:34 AM
To: users@kafka.apache.org
Cc: Neha Narkhede
Subject: Re: BadVersion state in Kafka Logs
Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you could make a patch that does would be great.
I will kick off a discussion for KAFKA-1382 and the scala 2.11 for 0.8.1.2 release (and see what others may think we should do like the gradle changes I think we should do too for src release issues (and the jars in the repo)). I will send that on dev/user in a little bit (please comment +1 community support please on that thread for the release).
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> I would love to help you guys to make Kafka best in Pub/Sub, will
> continue doing that whenever I can.
>
> Do we have 0.8.1.2 release tag or should we apply patch on top of
> 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
>
> Balaji
>
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Monday, September 29, 2014 5:21 PM
> To: Seshadri, Balaji
> Cc: users@kafka.apache.org
> Subject: Re: BadVersion state in Kafka Logs
>
> It is difficult to predict an exact date. Though all the discussions
> of the progress and ETA are on the mailing list. You can follow the
> discussions to know the details and/or offer to help out on the
> outstanding issues.
>
> On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> Neha,
>
> Do you know the date in Oct when 0.8.2 is going to be out ?.
>
> Thanks,
>
> Balaji
>
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
> neha.narkhede@gmail.com>]
> Sent: Thursday, September 25, 2014 1:08 PM
> To: Seshadri, Balaji
> Cc: users@kafka.apache.org<ma...@kafka.apache.org>
>
> Subject: Re: BadVersion state in Kafka Logs
>
> We are close to the release. I'd probably expect 0.8.2 sometime in October.
>
> On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> Hi Neha,
>
> Do you know when are you guys releasing 0.8.2 ?.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
> Balaji.Seshadri@dish.com>]
> Sent: Thursday, September 25, 2014 9:41 AM
> To: users@kafka.apache.org<ma...@kafka.apache.org>
> Subject: RE: BadVersion state in Kafka Logs
>
> Thanks for the replay.
>
> Please let me know if we can use trunk as 0.8.2 is not yet released.
>
> Balaji
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
> neha.narkhede@gmail.com>]
> Sent: Wednesday, September 24, 2014 6:32 PM
> To: users@kafka.apache.org<ma...@kafka.apache.org>
> Subject: Re: BadVersion state in Kafka Logs
>
> From the logs you've attached, my guess is it's most likely due to
> KAFKA-1382.
>
> Thanks,
> Neha
>
> On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>
> > wrote:
>
> > Hi,
> >
> >
> >
> > We got the below error in our logs and our consumers stopped
> > consuming any data ?.It worked only after restart.
> >
> >
> >
> > We would like to confirm that it's because we are running with
> > 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
> >
> >
> >
> > Please let me know if it's this KAFKA-1382 causing the issue.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Balaji
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > *Cc:* Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Adding full MAA distro.
> >
> >
> >
> > DES Offshore looked in to the logs on kafka servers and seems like
> > the issue we encountered yesterday may be described in these
> > threads, please have a look -
> >
> >
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > fix/patch which is available in 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Sharma, Navdeep
> > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > *To:* Gulia, Vikram; #IT-MAD DES
> > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Hi Vikram,
> >
> >
> >
> > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> > found that the error occurred only on 23 Sept. This is not
> > historical as we checked last 4 days logs.
> >
> >
> >
> > It looks like that consumer got stopped on September 22 2014 for
> > Linux patching activty.MAA started consumer September 23 2014 at 1:00 AM.
> >
> >
> >
> > *Issue *in server log *"* *BadVersion for
> > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > present in previous 4 days logs.
> >
> > More detail of this error can be found at-
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > We are not sure about data loss in this scenario and working on this.
> >
> >
> >
> >
> >
> >
> >
> > Let us know if any concerns.
> >
> >
> >
> > [image: cid:image001.gif@01CF7B0A.03F21580]
> >
> > Navdeep Sharma
> > Developer - offshore, Middleware Applications & Development o India:
> > 0120-4532000 - 2234
> > c: +91-9911698102<tel:%2B91-9911698102>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > *To:* #IT-MAD DES
> > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > DES Offshore dev,
> >
> >
> >
> > Please work with MAA offshore to monitor the kafka broker as we had
> > this incident where lot of partitions went offline around 1.45 PM
> > MST and MAA has to restart the kafka servers. We may have lost
> > messages and we need to see if there is a way to figure out what was the impact.
> >
> >
> >
> > Also, check the logs for kafka servers and see if we can figure out
> > why did partitions go offline or are un-available? Let us know if
> > you find anything relevant.
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Nielsen, Andy
> > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > *To:* #IT-MAD DES; Gulia, Vikram
> > *Cc:* #IT-MAA
> > *Subject:* 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic
> > --unavailable-partitions
> >
> > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-activated partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 43 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 73 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 103 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 27 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 57 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 87 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 117 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-info-updated partition: 7 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 37 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 67 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-info-updated partition: 97 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 127 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 21 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 51 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 81 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 111 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: dish-promo-application-access partition: 23 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 53 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 83 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 113 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: online-account-registration-attempted partition: 21 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 51 leader:
> > -1 replicas: 4,6,1 isr: 1
> >
> > topic: online-account-registration-attempted partition: 81 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 111 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 4 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 34 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 64 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 94 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-communications partition: 124 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 15 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 45 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 75 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 105 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: threshold-exceeded partition: 14 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 44 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 74 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 104 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> >
> >
> > *Andy Nielsen*
> >
> > *Middleware Application Admin*
> >
> > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
> >
> > *cell:720-971-2856<tel:720-971-2856>
> > <720-971-2856<tel:720-971-2856>>*
> >
> >
> >
>
>
>
Re: BadVersion state in Kafka Logs
Posted by Joe Stein <jo...@stealth.ly>.
Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you
could make a patch that does would be great.
I will kick off a discussion for KAFKA-1382 and the scala 2.11 for 0.8.1.2
release (and see what others may think we should do like the gradle changes
I think we should do too for src release issues (and the jars in the
repo)). I will send that on dev/user in a little bit (please comment +1
community support please on that thread for the release).
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> I would love to help you guys to make Kafka best in Pub/Sub, will continue
> doing that whenever I can.
>
> Do we have 0.8.1.2 release tag or should we apply patch on top of 0.8.1.1
> tag because we need this KAFKA-1382 JIRA ?.
>
> Balaji
>
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Monday, September 29, 2014 5:21 PM
> To: Seshadri, Balaji
> Cc: users@kafka.apache.org
> Subject: Re: BadVersion state in Kafka Logs
>
> It is difficult to predict an exact date. Though all the discussions of
> the progress and ETA are on the mailing list. You can follow the
> discussions to know the details and/or offer to help out on the outstanding
> issues.
>
> On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> Neha,
>
> Do you know the date in Oct when 0.8.2 is going to be out ?.
>
> Thanks,
>
> Balaji
>
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com<mailto:
> neha.narkhede@gmail.com>]
> Sent: Thursday, September 25, 2014 1:08 PM
> To: Seshadri, Balaji
> Cc: users@kafka.apache.org<ma...@kafka.apache.org>
>
> Subject: Re: BadVersion state in Kafka Logs
>
> We are close to the release. I'd probably expect 0.8.2 sometime in October.
>
> On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>> wrote:
> Hi Neha,
>
> Do you know when are you guys releasing 0.8.2 ?.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<mailto:
> Balaji.Seshadri@dish.com>]
> Sent: Thursday, September 25, 2014 9:41 AM
> To: users@kafka.apache.org<ma...@kafka.apache.org>
> Subject: RE: BadVersion state in Kafka Logs
>
> Thanks for the replay.
>
> Please let me know if we can use trunk as 0.8.2 is not yet released.
>
> Balaji
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com<mailto:
> neha.narkhede@gmail.com>]
> Sent: Wednesday, September 24, 2014 6:32 PM
> To: users@kafka.apache.org<ma...@kafka.apache.org>
> Subject: Re: BadVersion state in Kafka Logs
>
> From the logs you've attached, my guess is it's most likely due to
> KAFKA-1382.
>
> Thanks,
> Neha
>
> On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com<ma...@dish.com>
> > wrote:
>
> > Hi,
> >
> >
> >
> > We got the below error in our logs and our consumers stopped consuming
> > any data ?.It worked only after restart.
> >
> >
> >
> > We would like to confirm that it's because we are running with
> > 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
> >
> >
> >
> > Please let me know if it's this KAFKA-1382 causing the issue.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Balaji
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > *Cc:* Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Adding full MAA distro.
> >
> >
> >
> > DES Offshore looked in to the logs on kafka servers and seems like the
> > issue we encountered yesterday may be described in these threads,
> > please have a look -
> >
> >
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > fix/patch which is available in 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Sharma, Navdeep
> > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > *To:* Gulia, Vikram; #IT-MAD DES
> > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Hi Vikram,
> >
> >
> >
> > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> > found that the error occurred only on 23 Sept. This is not historical
> > as we checked last 4 days logs.
> >
> >
> >
> > It looks like that consumer got stopped on September 22 2014 for Linux
> > patching activty.MAA started consumer September 23 2014 at 1:00 AM.
> >
> >
> >
> > *Issue *in server log *"* *BadVersion for
> > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > present in previous 4 days logs.
> >
> > More detail of this error can be found at-
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > We are not sure about data loss in this scenario and working on this.
> >
> >
> >
> >
> >
> >
> >
> > Let us know if any concerns.
> >
> >
> >
> > [image: cid:image001.gif@01CF7B0A.03F21580]
> >
> > Navdeep Sharma
> > Developer - offshore, Middleware Applications & Development o India:
> > 0120-4532000 - 2234
> > c: +91-9911698102<tel:%2B91-9911698102>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > *To:* #IT-MAD DES
> > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > DES Offshore dev,
> >
> >
> >
> > Please work with MAA offshore to monitor the kafka broker as we had
> > this incident where lot of partitions went offline around 1.45 PM MST
> > and MAA has to restart the kafka servers. We may have lost messages
> > and we need to see if there is a way to figure out what was the impact.
> >
> >
> >
> > Also, check the logs for kafka servers and see if we can figure out
> > why did partitions go offline or are un-available? Let us know if you
> > find anything relevant.
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Nielsen, Andy
> > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > *To:* #IT-MAD DES; Gulia, Vikram
> > *Cc:* #IT-MAA
> > *Subject:* 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
> >
> > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-activated partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 43 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 73 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 103 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 27 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 57 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 87 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 117 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-info-updated partition: 7 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 37 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 67 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-info-updated partition: 97 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 127 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 21 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 51 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 81 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 111 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: dish-promo-application-access partition: 23 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 53 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 83 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 113 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: online-account-registration-attempted partition: 21 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 51 leader:
> > -1 replicas: 4,6,1 isr: 1
> >
> > topic: online-account-registration-attempted partition: 81 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 111 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 4 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 34 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 64 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 94 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-communications partition: 124 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 15 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 45 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 75 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 105 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: threshold-exceeded partition: 14 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 44 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 74 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 104 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> >
> >
> > *Andy Nielsen*
> >
> > *Middleware Application Admin*
> >
> > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
> >
> > *cell:720-971-2856<tel:720-971-2856> <720-971-2856<tel:720-971-2856>>*
> >
> >
> >
>
>
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
I would love to help you guys to make Kafka best in Pub/Sub, will continue doing that whenever I can.
Do we have 0.8.1.2 release tag or should we apply patch on top of 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?.
Balaji
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Monday, September 29, 2014 5:21 PM
To: Seshadri, Balaji
Cc: users@kafka.apache.org
Subject: Re: BadVersion state in Kafka Logs
It is difficult to predict an exact date. Though all the discussions of the progress and ETA are on the mailing list. You can follow the discussions to know the details and/or offer to help out on the outstanding issues.
On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <Ba...@dish.com>> wrote:
Neha,
Do you know the date in Oct when 0.8.2 is going to be out ?.
Thanks,
Balaji
From: Neha Narkhede [mailto:neha.narkhede@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 25, 2014 1:08 PM
To: Seshadri, Balaji
Cc: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Re: BadVersion state in Kafka Logs
We are close to the release. I'd probably expect 0.8.2 sometime in October.
On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <Ba...@dish.com>> wrote:
Hi Neha,
Do you know when are you guys releasing 0.8.2 ?.
Thanks,
Balaji
-----Original Message-----
From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<ma...@dish.com>]
Sent: Thursday, September 25, 2014 9:41 AM
To: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: RE: BadVersion state in Kafka Logs
Thanks for the replay.
Please let me know if we can use trunk as 0.8.2 is not yet released.
Balaji
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 24, 2014 6:32 PM
To: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Re: BadVersion state in Kafka Logs
From the logs you've attached, my guess is it's most likely due to KAFKA-1382.
Thanks,
Neha
On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <Ba...@dish.com>
> wrote:
> Hi,
>
>
>
> We got the below error in our logs and our consumers stopped consuming
> any data ?.It worked only after restart.
>
>
>
> We would like to confirm that it's because we are running with
> 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
>
>
>
> Please let me know if it's this KAFKA-1382 causing the issue.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Wednesday, September 24, 2014 8:43 AM
> *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> *Cc:* Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Adding full MAA distro.
>
>
>
> DES Offshore looked in to the logs on kafka servers and seems like the
> issue we encountered yesterday may be described in these threads,
> please have a look -
>
>
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> fix/patch which is available in 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Sharma, Navdeep
> *Sent:* Wednesday, September 24, 2014 6:53 AM
> *To:* Gulia, Vikram; #IT-MAD DES
> *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Hi Vikram,
>
>
>
> We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> found that the error occurred only on 23 Sept. This is not historical
> as we checked last 4 days logs.
>
>
>
> It looks like that consumer got stopped on September 22 2014 for Linux
> patching activty.MAA started consumer September 23 2014 at 1:00 AM.
>
>
>
> *Issue *in server log *"* *BadVersion for
> /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> present in previous 4 days logs.
>
> More detail of this error can be found at-
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> We are not sure about data loss in this scenario and working on this.
>
>
>
>
>
>
>
> Let us know if any concerns.
>
>
>
> [image: cid:image001.gif@01CF7B0A.03F21580]
>
> Navdeep Sharma
> Developer - offshore, Middleware Applications & Development o India:
> 0120-4532000 - 2234
> c: +91-9911698102<tel:%2B91-9911698102>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Tuesday, September 23, 2014 6:17 PM
> *To:* #IT-MAD DES
> *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>
>
>
> DES Offshore dev,
>
>
>
> Please work with MAA offshore to monitor the kafka broker as we had
> this incident where lot of partitions went offline around 1.45 PM MST
> and MAA has to restart the kafka servers. We may have lost messages
> and we need to see if there is a way to figure out what was the impact.
>
>
>
> Also, check the logs for kafka servers and see if we can figure out
> why did partitions go offline or are un-available? Let us know if you
> find anything relevant.
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Nielsen, Andy
> *Sent:* Tuesday, September 23, 2014 5:04 PM
> *To:* #IT-MAD DES; Gulia, Vikram
> *Cc:* #IT-MAA
> *Subject:* 9/23 prod issue - offline kafka partitions.
>
>
>
> desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
>
> topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-activated partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 43 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 73 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 103 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 27 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 57 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 87 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 117 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-info-updated partition: 7 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 37 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 67 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-info-updated partition: 97 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 127 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 21 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 51 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 81 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 111 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: dish-promo-application-access partition: 23 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 53 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 83 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 113 leader: -1
> replicas: 4,6,1 isr:
>
> topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: online-account-registration-attempted partition: 21 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 51 leader:
> -1 replicas: 4,6,1 isr: 1
>
> topic: online-account-registration-attempted partition: 81 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 111 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 4 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 34 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 64 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 94 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-communications partition: 124 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 15 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 45 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 75 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 105 leader: -1
> replicas: 4,6,1 isr:
>
> topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> isr:
>
> topic: threshold-exceeded partition: 14 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 44 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 74 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 104 leader: -1 replicas:
> 4,6,1 isr: 1
>
>
>
> *Andy Nielsen*
>
> *Middleware Application Admin*
>
> *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>*
>
> *cell:720-971-2856<tel:720-971-2856> <720-971-2856<tel:720-971-2856>>*
>
>
>
Re: BadVersion state in Kafka Logs
Posted by Neha Narkhede <ne...@gmail.com>.
It is difficult to predict an exact date. Though all the discussions of the
progress and ETA are on the mailing list. You can follow the discussions to
know the details and/or offer to help out on the outstanding issues.
On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji <Ba...@dish.com>
wrote:
> Neha,
>
>
>
> Do you know the date in Oct when 0.8.2 is going to be out ?.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Neha Narkhede [mailto:neha.narkhede@gmail.com]
> *Sent:* Thursday, September 25, 2014 1:08 PM
> *To:* Seshadri, Balaji
> *Cc:* users@kafka.apache.org
>
> *Subject:* Re: BadVersion state in Kafka Logs
>
>
>
> We are close to the release. I'd probably expect 0.8.2 sometime in October.
>
>
>
> On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com> wrote:
>
> Hi Neha,
>
> Do you know when are you guys releasing 0.8.2 ?.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
>
> Sent: Thursday, September 25, 2014 9:41 AM
> To: users@kafka.apache.org
> Subject: RE: BadVersion state in Kafka Logs
>
> Thanks for the replay.
>
> Please let me know if we can use trunk as 0.8.2 is not yet released.
>
> Balaji
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com]
> Sent: Wednesday, September 24, 2014 6:32 PM
> To: users@kafka.apache.org
> Subject: Re: BadVersion state in Kafka Logs
>
> From the logs you've attached, my guess is it's most likely due to
> KAFKA-1382.
>
> Thanks,
> Neha
>
> On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > Hi,
> >
> >
> >
> > We got the below error in our logs and our consumers stopped consuming
> > any data ?.It worked only after restart.
> >
> >
> >
> > We would like to confirm that it's because we are running with
> > 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
> >
> >
> >
> > Please let me know if it's this KAFKA-1382 causing the issue.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Balaji
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > *Cc:* Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Adding full MAA distro.
> >
> >
> >
> > DES Offshore looked in to the logs on kafka servers and seems like the
> > issue we encountered yesterday may be described in these threads,
> > please have a look -
> >
> >
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > fix/patch which is available in 0.8.1.2/0.8.2)
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Sharma, Navdeep
> > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > *To:* Gulia, Vikram; #IT-MAD DES
> > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Hi Vikram,
> >
> >
> >
> > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> > found that the error occurred only on 23 Sept. This is not historical
> > as we checked last 4 days logs.
> >
> >
> >
> > It looks like that consumer got stopped on September 22 2014 for Linux
> > patching activty.MAA started consumer September 23 2014 at 1:00 AM.
> >
> >
> >
> > *Issue *in server log *"* *BadVersion for
> > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > present in previous 4 days logs.
> >
> > More detail of this error can be found at-
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > We are not sure about data loss in this scenario and working on this.
> >
> >
> >
> >
> >
> >
> >
> > Let us know if any concerns.
> >
> >
> >
> > [image: cid:image001.gif@01CF7B0A.03F21580]
> >
> > Navdeep Sharma
> > Developer - offshore, Middleware Applications & Development o India:
> > 0120-4532000 - 2234
> > c: +91-9911698102
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > *To:* #IT-MAD DES
> > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > DES Offshore dev,
> >
> >
> >
> > Please work with MAA offshore to monitor the kafka broker as we had
> > this incident where lot of partitions went offline around 1.45 PM MST
> > and MAA has to restart the kafka servers. We may have lost messages
> > and we need to see if there is a way to figure out what was the impact.
> >
> >
> >
> > Also, check the logs for kafka servers and see if we can figure out
> > why did partitions go offline or are un-available? Let us know if you
> > find anything relevant.
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Nielsen, Andy
> > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > *To:* #IT-MAD DES; Gulia, Vikram
> > *Cc:* #IT-MAA
> > *Subject:* 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
> >
> > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-activated partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 43 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 73 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 103 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 27 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 57 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 87 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 117 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-info-updated partition: 7 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 37 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 67 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-info-updated partition: 97 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 127 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 21 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 51 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 81 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 111 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: dish-promo-application-access partition: 23 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 53 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 83 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 113 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: online-account-registration-attempted partition: 21 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 51 leader:
> > -1 replicas: 4,6,1 isr: 1
> >
> > topic: online-account-registration-attempted partition: 81 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 111 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 4 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 34 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 64 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 94 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-communications partition: 124 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 15 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 45 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 75 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 105 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: threshold-exceeded partition: 14 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 44 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 74 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 104 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> >
> >
> > *Andy Nielsen*
> >
> > *Middleware Application Admin*
> >
> > *303-723-2347 <303-723-2347>*
> >
> > *cell:720-971-2856 <720-971-2856>*
> >
> >
> >
>
>
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
Neha,
Do you know the date in Oct when 0.8.2 is going to be out ?.
Thanks,
Balaji
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Thursday, September 25, 2014 1:08 PM
To: Seshadri, Balaji
Cc: users@kafka.apache.org
Subject: Re: BadVersion state in Kafka Logs
We are close to the release. I'd probably expect 0.8.2 sometime in October.
On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <Ba...@dish.com>> wrote:
Hi Neha,
Do you know when are you guys releasing 0.8.2 ?.
Thanks,
Balaji
-----Original Message-----
From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com<ma...@dish.com>]
Sent: Thursday, September 25, 2014 9:41 AM
To: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: RE: BadVersion state in Kafka Logs
Thanks for the replay.
Please let me know if we can use trunk as 0.8.2 is not yet released.
Balaji
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 24, 2014 6:32 PM
To: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Re: BadVersion state in Kafka Logs
From the logs you've attached, my guess is it's most likely due to KAFKA-1382.
Thanks,
Neha
On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <Ba...@dish.com>
> wrote:
> Hi,
>
>
>
> We got the below error in our logs and our consumers stopped consuming
> any data ?.It worked only after restart.
>
>
>
> We would like to confirm that it's because we are running with
> 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
>
>
>
> Please let me know if it's this KAFKA-1382 causing the issue.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Wednesday, September 24, 2014 8:43 AM
> *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> *Cc:* Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Adding full MAA distro.
>
>
>
> DES Offshore looked in to the logs on kafka servers and seems like the
> issue we encountered yesterday may be described in these threads,
> please have a look -
>
>
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> fix/patch which is available in 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>)
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Sharma, Navdeep
> *Sent:* Wednesday, September 24, 2014 6:53 AM
> *To:* Gulia, Vikram; #IT-MAD DES
> *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Hi Vikram,
>
>
>
> We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> found that the error occurred only on 23 Sept. This is not historical
> as we checked last 4 days logs.
>
>
>
> It looks like that consumer got stopped on September 22 2014 for Linux
> patching activty.MAA started consumer September 23 2014 at 1:00 AM.
>
>
>
> *Issue *in server log *"* *BadVersion for
> /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> present in previous 4 days logs.
>
> More detail of this error can be found at-
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> We are not sure about data loss in this scenario and working on this.
>
>
>
>
>
>
>
> Let us know if any concerns.
>
>
>
> [image: cid:image001.gif@01CF7B0A.03F21580]
>
> Navdeep Sharma
> Developer - offshore, Middleware Applications & Development o India:
> 0120-4532000 - 2234
> c: +91-9911698102<tel:%2B91-9911698102>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Tuesday, September 23, 2014 6:17 PM
> *To:* #IT-MAD DES
> *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>
>
>
> DES Offshore dev,
>
>
>
> Please work with MAA offshore to monitor the kafka broker as we had
> this incident where lot of partitions went offline around 1.45 PM MST
> and MAA has to restart the kafka servers. We may have lost messages
> and we need to see if there is a way to figure out what was the impact.
>
>
>
> Also, check the logs for kafka servers and see if we can figure out
> why did partitions go offline or are un-available? Let us know if you
> find anything relevant.
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Nielsen, Andy
> *Sent:* Tuesday, September 23, 2014 5:04 PM
> *To:* #IT-MAD DES; Gulia, Vikram
> *Cc:* #IT-MAA
> *Subject:* 9/23 prod issue - offline kafka partitions.
>
>
>
> desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
>
> topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-activated partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 43 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 73 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 103 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 27 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 57 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 87 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 117 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-info-updated partition: 7 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 37 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 67 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-info-updated partition: 97 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 127 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 21 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 51 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 81 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 111 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: dish-promo-application-access partition: 23 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 53 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 83 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 113 leader: -1
> replicas: 4,6,1 isr:
>
> topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: online-account-registration-attempted partition: 21 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 51 leader:
> -1 replicas: 4,6,1 isr: 1
>
> topic: online-account-registration-attempted partition: 81 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 111 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 4 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 34 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 64 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 94 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-communications partition: 124 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 15 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 45 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 75 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 105 leader: -1
> replicas: 4,6,1 isr:
>
> topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> isr:
>
> topic: threshold-exceeded partition: 14 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 44 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 74 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 104 leader: -1 replicas:
> 4,6,1 isr: 1
>
>
>
> *Andy Nielsen*
>
> *Middleware Application Admin*
>
> *303-723-2347 <303-723-2347>*
>
> *cell:720-971-2856 <720-971-2856>*
>
>
>
Re: BadVersion state in Kafka Logs
Posted by Neha Narkhede <ne...@gmail.com>.
We are close to the release. I'd probably expect 0.8.2 sometime in October.
On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi Neha,
>
> Do you know when are you guys releasing 0.8.2 ?.
>
> Thanks,
>
> Balaji
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> Sent: Thursday, September 25, 2014 9:41 AM
> To: users@kafka.apache.org
> Subject: RE: BadVersion state in Kafka Logs
>
> Thanks for the replay.
>
> Please let me know if we can use trunk as 0.8.2 is not yet released.
>
> Balaji
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com]
> Sent: Wednesday, September 24, 2014 6:32 PM
> To: users@kafka.apache.org
> Subject: Re: BadVersion state in Kafka Logs
>
> From the logs you've attached, my guess is it's most likely due to
> KAFKA-1382.
>
> Thanks,
> Neha
>
> On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <
> Balaji.Seshadri@dish.com
> > wrote:
>
> > Hi,
> >
> >
> >
> > We got the below error in our logs and our consumers stopped consuming
> > any data ?.It worked only after restart.
> >
> >
> >
> > We would like to confirm that it's because we are running with
> > 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
> >
> >
> >
> > Please let me know if it's this KAFKA-1382 causing the issue.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Balaji
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Wednesday, September 24, 2014 8:43 AM
> > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> > *Cc:* Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Adding full MAA distro.
> >
> >
> >
> > DES Offshore looked in to the logs on kafka servers and seems like the
> > issue we encountered yesterday may be described in these threads,
> > please have a look -
> >
> >
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> > fix/patch which is available in 0.8.1.2/0.8.2)
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Sharma, Navdeep
> > *Sent:* Wednesday, September 24, 2014 6:53 AM
> > *To:* Gulia, Vikram; #IT-MAD DES
> > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> > *Subject:* RE: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > Hi Vikram,
> >
> >
> >
> > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> > found that the error occurred only on 23 Sept. This is not historical
> > as we checked last 4 days logs.
> >
> >
> >
> > It looks like that consumer got stopped on September 22 2014 for Linux
> > patching activty.MAA started consumer September 23 2014 at 1:00 AM.
> >
> >
> >
> > *Issue *in server log *"* *BadVersion for
> > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> > present in previous 4 days logs.
> >
> > More detail of this error can be found at-
> >
> > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
> >
> >
> >
> > We are not sure about data loss in this scenario and working on this.
> >
> >
> >
> >
> >
> >
> >
> > Let us know if any concerns.
> >
> >
> >
> > [image: cid:image001.gif@01CF7B0A.03F21580]
> >
> > Navdeep Sharma
> > Developer - offshore, Middleware Applications & Development o India:
> > 0120-4532000 - 2234
> > c: +91-9911698102
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *From:* Gulia, Vikram
> > *Sent:* Tuesday, September 23, 2014 6:17 PM
> > *To:* #IT-MAD DES
> > *Subject:* FW: 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > DES Offshore dev,
> >
> >
> >
> > Please work with MAA offshore to monitor the kafka broker as we had
> > this incident where lot of partitions went offline around 1.45 PM MST
> > and MAA has to restart the kafka servers. We may have lost messages
> > and we need to see if there is a way to figure out what was the impact.
> >
> >
> >
> > Also, check the logs for kafka servers and see if we can figure out
> > why did partitions go offline or are un-available? Let us know if you
> > find anything relevant.
> >
> >
> >
> > Thank You,
> >
> > Vikram Gulia
> >
> >
> >
> > *From:* Nielsen, Andy
> > *Sent:* Tuesday, September 23, 2014 5:04 PM
> > *To:* #IT-MAD DES; Gulia, Vikram
> > *Cc:* #IT-MAA
> > *Subject:* 9/23 prod issue - offline kafka partitions.
> >
> >
> >
> > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
> >
> > topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: account-activated partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 43 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 73 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-activated partition: 103 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 27 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 57 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-adjustment-issued partition: 87 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: account-adjustment-issued partition: 117 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: account-info-updated partition: 7 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 37 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 67 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: account-info-updated partition: 97 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: account-info-updated partition: 127 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 21 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 51 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 81 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: application-access partition: 111 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: dish-promo-application-access partition: 23 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 53 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 83 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: dish-promo-application-access partition: 113 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: online-account-registration-attempted partition: 21 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 51 leader:
> > -1 replicas: 4,6,1 isr: 1
> >
> > topic: online-account-registration-attempted partition: 81 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: online-account-registration-attempted partition: 111 leader:
> > -1 replicas: 4,6,1 isr:
> >
> > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 4 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 34 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 64 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: outbound-communications partition: 94 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: outbound-communications partition: 124 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 15 leader: -1
> > replicas: 4,6,1 isr: 1
> >
> > topic: postal-mail-undeliverable partition: 45 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 75 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: postal-mail-undeliverable partition: 105 leader: -1
> > replicas: 4,6,1 isr:
> >
> > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> > isr: 1
> >
> > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> > isr:
> >
> > topic: threshold-exceeded partition: 14 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 44 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 74 leader: -1 replicas:
> > 4,6,1 isr:
> >
> > topic: threshold-exceeded partition: 104 leader: -1 replicas:
> > 4,6,1 isr: 1
> >
> >
> >
> > *Andy Nielsen*
> >
> > *Middleware Application Admin*
> >
> > *303-723-2347 <303-723-2347>*
> >
> > *cell:720-971-2856 <720-971-2856>*
> >
> >
> >
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
Hi Neha,
Do you know when are you guys releasing 0.8.2 ?.
Thanks,
Balaji
-----Original Message-----
From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
Sent: Thursday, September 25, 2014 9:41 AM
To: users@kafka.apache.org
Subject: RE: BadVersion state in Kafka Logs
Thanks for the replay.
Please let me know if we can use trunk as 0.8.2 is not yet released.
Balaji
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Wednesday, September 24, 2014 6:32 PM
To: users@kafka.apache.org
Subject: Re: BadVersion state in Kafka Logs
>From the logs you've attached, my guess is it's most likely due to KAFKA-1382.
Thanks,
Neha
On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi,
>
>
>
> We got the below error in our logs and our consumers stopped consuming
> any data ?.It worked only after restart.
>
>
>
> We would like to confirm that it's because we are running with
> 0.8-beta version and not 0.8 release version to convince "THE MGMT" guys.
>
>
>
> Please let me know if it's this KAFKA-1382 causing the issue.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Wednesday, September 24, 2014 8:43 AM
> *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> *Cc:* Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Adding full MAA distro.
>
>
>
> DES Offshore looked in to the logs on kafka servers and seems like the
> issue we encountered yesterday may be described in these threads,
> please have a look -
>
>
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> fix/patch which is available in 0.8.1.2/0.8.2)
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Sharma, Navdeep
> *Sent:* Wednesday, September 24, 2014 6:53 AM
> *To:* Gulia, Vikram; #IT-MAD DES
> *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Hi Vikram,
>
>
>
> We analyzed below mentioned issue with MAA-Offshore (Abhishek) and
> found that the error occurred only on 23 Sept. This is not historical
> as we checked last 4 days logs.
>
>
>
> It looks like that consumer got stopped on September 22 2014 for Linux
> patching activty.MAA started consumer September 23 2014 at 1:00 AM.
>
>
>
> *Issue *in server log *"* *BadVersion for
> /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> present in previous 4 days logs.
>
> More detail of this error can be found at-
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> We are not sure about data loss in this scenario and working on this.
>
>
>
>
>
>
>
> Let us know if any concerns.
>
>
>
> [image: cid:image001.gif@01CF7B0A.03F21580]
>
> Navdeep Sharma
> Developer - offshore, Middleware Applications & Development o India:
> 0120-4532000 - 2234
> c: +91-9911698102
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Tuesday, September 23, 2014 6:17 PM
> *To:* #IT-MAD DES
> *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>
>
>
> DES Offshore dev,
>
>
>
> Please work with MAA offshore to monitor the kafka broker as we had
> this incident where lot of partitions went offline around 1.45 PM MST
> and MAA has to restart the kafka servers. We may have lost messages
> and we need to see if there is a way to figure out what was the impact.
>
>
>
> Also, check the logs for kafka servers and see if we can figure out
> why did partitions go offline or are un-available? Let us know if you
> find anything relevant.
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Nielsen, Andy
> *Sent:* Tuesday, September 23, 2014 5:04 PM
> *To:* #IT-MAD DES; Gulia, Vikram
> *Cc:* #IT-MAA
> *Subject:* 9/23 prod issue - offline kafka partitions.
>
>
>
> desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
>
> topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-activated partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 43 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 73 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 103 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 27 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 57 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 87 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 117 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-info-updated partition: 7 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 37 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 67 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-info-updated partition: 97 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 127 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 21 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 51 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 81 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 111 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: dish-promo-application-access partition: 23 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 53 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 83 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 113 leader: -1
> replicas: 4,6,1 isr:
>
> topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: online-account-registration-attempted partition: 21 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 51 leader:
> -1 replicas: 4,6,1 isr: 1
>
> topic: online-account-registration-attempted partition: 81 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 111 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 4 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 34 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 64 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 94 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-communications partition: 124 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 15 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 45 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 75 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 105 leader: -1
> replicas: 4,6,1 isr:
>
> topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> isr:
>
> topic: threshold-exceeded partition: 14 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 44 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 74 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 104 leader: -1 replicas:
> 4,6,1 isr: 1
>
>
>
> *Andy Nielsen*
>
> *Middleware Application Admin*
>
> *303-723-2347 <303-723-2347>*
>
> *cell:720-971-2856 <720-971-2856>*
>
>
>
RE: BadVersion state in Kafka Logs
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
Thanks for the replay.
Please let me know if we can use trunk as 0.8.2 is not yet released.
Balaji
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Wednesday, September 24, 2014 6:32 PM
To: users@kafka.apache.org
Subject: Re: BadVersion state in Kafka Logs
>From the logs you've attached, my guess is it's most likely due to
KAFKA-1382.
Thanks,
Neha
On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi,
>
>
>
> We got the below error in our logs and our consumers stopped consuming any
> data ?.It worked only after restart.
>
>
>
> We would like to confirm that it’s because we are running with 0.8-beta
> version and not 0.8 release version to convince “THE MGMT” guys.
>
>
>
> Please let me know if it’s this KAFKA-1382 causing the issue.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Wednesday, September 24, 2014 8:43 AM
> *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> *Cc:* Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Adding full MAA distro.
>
>
>
> DES Offshore looked in to the logs on kafka servers and seems like the
> issue we encountered yesterday may be described in these threads, please
> have a look -
>
>
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> fix/patch which is available in 0.8.1.2/0.8.2)
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Sharma, Navdeep
> *Sent:* Wednesday, September 24, 2014 6:53 AM
> *To:* Gulia, Vikram; #IT-MAD DES
> *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Hi Vikram,
>
>
>
> We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found
> that the error occurred only on 23 Sept. This is not historical as we
> checked last 4 days logs.
>
>
>
> It looks like that consumer got stopped on September 22 2014 for Linux
> patching activty.MAA started consumer September 23 2014 at 1:00 AM.
>
>
>
> *Issue *in server log *"* *BadVersion for
> /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> present in previous 4 days logs.
>
> More detail of this error can be found at-
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> We are not sure about data loss in this scenario and working on this.
>
>
>
>
>
>
>
> Let us know if any concerns.
>
>
>
> [image: cid:image001.gif@01CF7B0A.03F21580]
>
> Navdeep Sharma
> Developer - offshore, Middleware Applications & Development
> o India: 0120-4532000 - 2234
> c: +91-9911698102
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Tuesday, September 23, 2014 6:17 PM
> *To:* #IT-MAD DES
> *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>
>
>
> DES Offshore dev,
>
>
>
> Please work with MAA offshore to monitor the kafka broker as we had this
> incident where lot of partitions went offline around 1.45 PM MST and MAA
> has to restart the kafka servers. We may have lost messages and we need to
> see if there is a way to figure out what was the impact.
>
>
>
> Also, check the logs for kafka servers and see if we can figure out why
> did partitions go offline or are un-available? Let us know if you find
> anything relevant.
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Nielsen, Andy
> *Sent:* Tuesday, September 23, 2014 5:04 PM
> *To:* #IT-MAD DES; Gulia, Vikram
> *Cc:* #IT-MAA
> *Subject:* 9/23 prod issue - offline kafka partitions.
>
>
>
> desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
>
> topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-activated partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 43 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 73 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 103 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 27 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 57 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 87 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 117 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-info-updated partition: 7 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 37 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 67 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-info-updated partition: 97 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 127 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 21 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 51 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 81 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 111 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: dish-promo-application-access partition: 23 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 53 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 83 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 113 leader: -1
> replicas: 4,6,1 isr:
>
> topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: online-account-registration-attempted partition: 21 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 51 leader:
> -1 replicas: 4,6,1 isr: 1
>
> topic: online-account-registration-attempted partition: 81 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 111 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 4 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 34 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 64 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 94 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-communications partition: 124 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 15 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 45 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 75 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 105 leader: -1
> replicas: 4,6,1 isr:
>
> topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> isr:
>
> topic: threshold-exceeded partition: 14 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 44 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 74 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 104 leader: -1 replicas:
> 4,6,1 isr: 1
>
>
>
> *Andy Nielsen*
>
> *Middleware Application Admin*
>
> *303-723-2347 <303-723-2347>*
>
> *cell:720-971-2856 <720-971-2856>*
>
>
>
Re: BadVersion state in Kafka Logs
Posted by Neha Narkhede <ne...@gmail.com>.
>From the logs you've attached, my guess is it's most likely due to
KAFKA-1382.
Thanks,
Neha
On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <Balaji.Seshadri@dish.com
> wrote:
> Hi,
>
>
>
> We got the below error in our logs and our consumers stopped consuming any
> data ?.It worked only after restart.
>
>
>
> We would like to confirm that it’s because we are running with 0.8-beta
> version and not 0.8 release version to convince “THE MGMT” guys.
>
>
>
> Please let me know if it’s this KAFKA-1382 causing the issue.
>
>
>
> Thanks,
>
>
>
> Balaji
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Wednesday, September 24, 2014 8:43 AM
> *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA
> *Cc:* Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Adding full MAA distro.
>
>
>
> DES Offshore looked in to the logs on kafka servers and seems like the
> issue we encountered yesterday may be described in these threads, please
> have a look -
>
>
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the
> fix/patch which is available in 0.8.1.2/0.8.2)
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Sharma, Navdeep
> *Sent:* Wednesday, September 24, 2014 6:53 AM
> *To:* Gulia, Vikram; #IT-MAD DES
> *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah
> *Subject:* RE: 9/23 prod issue - offline kafka partitions.
>
>
>
> Hi Vikram,
>
>
>
> We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found
> that the error occurred only on 23 Sept. This is not historical as we
> checked last 4 days logs.
>
>
>
> It looks like that consumer got stopped on September 22 2014 for Linux
> patching activty.MAA started consumer September 23 2014 at 1:00 AM.
>
>
>
> *Issue *in server log *"* *BadVersion for
> /brokers/topics/rain-burn-in/partitions/121/state"* but it is not
> present in previous 4 days logs.
>
> More detail of this error can be found at-
>
> http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
>
>
>
> We are not sure about data loss in this scenario and working on this.
>
>
>
>
>
>
>
> Let us know if any concerns.
>
>
>
> [image: cid:image001.gif@01CF7B0A.03F21580]
>
> Navdeep Sharma
> Developer - offshore, Middleware Applications & Development
> o India: 0120-4532000 - 2234
> c: +91-9911698102
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Gulia, Vikram
> *Sent:* Tuesday, September 23, 2014 6:17 PM
> *To:* #IT-MAD DES
> *Subject:* FW: 9/23 prod issue - offline kafka partitions.
>
>
>
> DES Offshore dev,
>
>
>
> Please work with MAA offshore to monitor the kafka broker as we had this
> incident where lot of partitions went offline around 1.45 PM MST and MAA
> has to restart the kafka servers. We may have lost messages and we need to
> see if there is a way to figure out what was the impact.
>
>
>
> Also, check the logs for kafka servers and see if we can figure out why
> did partitions go offline or are un-available? Let us know if you find
> anything relevant.
>
>
>
> Thank You,
>
> Vikram Gulia
>
>
>
> *From:* Nielsen, Andy
> *Sent:* Tuesday, September 23, 2014 5:04 PM
> *To:* #IT-MAD DES; Gulia, Vikram
> *Cc:* #IT-MAA
> *Subject:* 9/23 prod issue - offline kafka partitions.
>
>
>
> desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
>
> topic: account-access partition: 21 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 51 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-access partition: 81 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-access partition: 111 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: account-activated partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 43 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 73 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-activated partition: 103 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 27 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 57 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-adjustment-issued partition: 87 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: account-adjustment-issued partition: 117 leader: -1
> replicas: 4,6,1 isr:
>
> topic: account-created partition: 11 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 41 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 71 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-created partition: 101 leader: -1 replicas: 4,6,1
> isr:
>
> topic: account-info-updated partition: 7 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 37 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 67 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: account-info-updated partition: 97 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: account-info-updated partition: 127 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 21 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 51 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 81 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: application-access partition: 111 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 29 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: collected-event partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: collected-event partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: dish-promo-application-access partition: 23 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 53 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 83 leader: -1
> replicas: 4,6,1 isr:
>
> topic: dish-promo-application-access partition: 113 leader: -1
> replicas: 4,6,1 isr:
>
> topic: event-response partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 32 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: event-response partition: 122 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: leads-service partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 84 leader: -1 replicas: 4,6,1
> isr:
>
> topic: leads-service partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1
> isr:
>
> topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: online-account-registration-attempted partition: 21 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 51 leader:
> -1 replicas: 4,6,1 isr: 1
>
> topic: online-account-registration-attempted partition: 81 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: online-account-registration-attempted partition: 111 leader:
> -1 replicas: 4,6,1 isr:
>
> topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-completed partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-completed partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 25 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 55 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 85 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-created partition: 115 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 8 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-modified partition: 38 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 98 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-modified partition: 128 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 24 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 54 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-request partition: 84 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-request partition: 114 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 27 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: order-response partition: 57 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 87 leader: -1 replicas: 4,6,1
> isr:
>
> topic: order-response partition: 117 leader: -1 replicas: 4,6,1
> isr:
>
> topic: outbound-call-attempted partition: 13 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-call-attempted partition: 43 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 73 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-call-attempted partition: 103 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 4 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 34 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 64 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: outbound-communications partition: 94 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: outbound-communications partition: 124 leader: -1 replicas:
> 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 15 leader: -1
> replicas: 4,6,1 isr: 1
>
> topic: postal-mail-undeliverable partition: 45 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 75 leader: -1
> replicas: 4,6,1 isr:
>
> topic: postal-mail-undeliverable partition: 105 leader: -1
> replicas: 4,6,1 isr:
>
> topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1
> isr:
>
> topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1
> isr: 1
>
> topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1
> isr:
>
> topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1
> isr:
>
> topic: threshold-exceeded partition: 14 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 44 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 74 leader: -1 replicas:
> 4,6,1 isr:
>
> topic: threshold-exceeded partition: 104 leader: -1 replicas:
> 4,6,1 isr: 1
>
>
>
> *Andy Nielsen*
>
> *Middleware Application Admin*
>
> *303-723-2347 <303-723-2347>*
>
> *cell:720-971-2856 <720-971-2856>*
>
>
>