You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mahesh Patade <Ma...@nw18.com> on 2017/07/31 17:44:56 UTC
Getting error while consuming data from broker
Hi All,
We are having 8 broker kafka cluster configured in our setup and created a topic with 8 partitions & 3 replicas. While trying to consume data from one broker(id:6) we are getting below errors and increase in lag for active partition on that host. We even tried restarting, deleting logs and reinstalling broker but still no luck. This is happening only for one broker. If we shut down that broker service everything works well.
Kafka Server Logs:
[2017-07-31 19:32:39,212] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-5 in 5 milliseconds. (kafka.coordinator.GroupMe
tadataManager)
[2017-07-31 19:32:39,217] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-13 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,218] INFO [Group Metadata Manager on Broker 6]: Loading offsets and group metadata from __consumer_offsets-13 (kafka.coordinator.GroupMetadataMa
nager)
[2017-07-31 19:32:39,220] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-37 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,230] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-45 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-13 in 13 milliseconds. (kafka.coordinator.Group
MetadataManager)
[2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]: Loading offsets and group metadata from __consumer_offsets-37 (kafka.coordinator.GroupMetadataMa
nager)
[2017-07-31 19:32:39,248] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions bseEquityAggFeedProd-4 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,249] INFO Truncating log bseEquityAggFeedProd-4 to offset 13082598. (kafka.log.Log)
[2017-07-31 19:32:39,276] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([bseEquityAggFeedProd-4, initOffset 13082598 to broker BrokerEnd
Point(2,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,277] INFO [ReplicaFetcherThread-0-2], Starting (kafka.server.ReplicaFetcherThread)
[2017-07-31 19:32:39,294] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-17 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,294] INFO Truncating log __consumer_offsets-17 to offset 0. (kafka.log.Log)
[2017-07-31 19:32:39,300] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([__consumer_offsets-17, initOffset 0 to broker BrokerEndPoint(2,
172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,304] ERROR [ReplicaFetcherThread-0-5], Error for partition [__consumer_offsets,9] to broker 5:org.apache.kafka.common.errors.NotLeaderForPartiti
onException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-07-31 19:32:39,308] ERROR [ReplicaFetcherThread-0-2], Error for partition [bseEquityAggFeedProd,4] to broker 2:org.apache.kafka.common.errors.NotLeaderForParti
tionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-07-31 19:32:39,313] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,17] to broker 2:org.apache.kafka.common.errors.NotLeaderForPartit
ionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-07-31 19:32:39,318] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-9 (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,319] INFO Truncating log __consumer_offsets-9 to offset 392. (kafka.log.Log)
[2017-07-31 19:32:39,324] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([__consumer_offsets-9, initOffset 392 to broker BrokerEndPoint(2
,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
[2017-07-31 19:32:39,339] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,9] to broker 2:org.apache.kafka.common.errors.NotLeaderForPartiti
onException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-07-31 19:32:39,986] INFO [GroupCoordinator 6]: Loading group metadata for bseEquityLegacyBod with generation 4 (kafka.coordinator.GroupCoordinator)
[2017-07-31 19:32:39,991] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-37 in 759 milliseconds. (kafka.coordinator.Grou
pMetadataManager)
Consumer Logs:
17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Discovered coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group bseEquityLegacyBod.
17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Marking the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for group bseEquityLegacyBod
17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Discovered coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group bseEquityLegacyBod.
17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Marking the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for group bseEquityLegacyBod
Thanks in advance.
Regards,
Mahesh Patade
________________________________
DISCLAIMER AND PRIVILEGE NOTICE : This e-mail and any files transmitted with it contain confidential, copyright, proprietary and legally privileged information. It should not be used by anyone who is not the original intended recipient. Any use, distribution, copying or disclosure by any other person is strictly prohibited. If you receive this transmission in error, please notify the sender by reply email and then destroy the message. Opinions, conclusions and other information in this message that do not relate to official business of Network18 Media & Investments Ltd., its subsidiaries, holding companies i.e. Network18 group shall be understood to be neither given nor endorsed by Network18 group. Internet communications cannot be guaranteed to be timely, Secure, error or virus-free. The sender does not accept liability for any errors or omissions.
Re: Getting error while consuming data from broker
Posted by "M. Manna" <ma...@gmail.com>.
Mahesh,
Thanks for sharing the info. Is having "Exactly" 8 brokers a "Must" for
you? because one of them is technically unnecessary since your cluster can
only tolerate 3 failures (even with 7 brokers).
Could you please try the following:
1) Stop the cluster.
2) Increase the number of renum.recovery.threads.per.data.dir from 1 to 5
(i mean something bigger than 1).
3) Retry creating a topic with both partitions and replication-factor set
to 8.
Let us know how it goes.
Regards,
On 31 July 2017 at 18:44, Mahesh Patade <Ma...@nw18.com> wrote:
> Hi All,
> We are having 8 broker kafka cluster configured in our setup and created a
> topic with 8 partitions & 3 replicas. While trying to consume data from one
> broker(id:6) we are getting below errors and increase in lag for active
> partition on that host. We even tried restarting, deleting logs and
> reinstalling broker but still no luck. This is happening only for one
> broker. If we shut down that broker service everything works well.
> Kafka Server Logs:
> [2017-07-31 19:32:39,212] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-5 in 5 milliseconds.
> (kafka.coordinator.GroupMe
> tadataManager)
> [2017-07-31 19:32:39,217] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-13 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,218] INFO [Group Metadata Manager on Broker 6]:
> Loading offsets and group metadata from __consumer_offsets-13
> (kafka.coordinator.GroupMetadataMa
> nager)
> [2017-07-31 19:32:39,220] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-37 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,230] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-45 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-13 in 13 milliseconds.
> (kafka.coordinator.Group
> MetadataManager)
> [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]:
> Loading offsets and group metadata from __consumer_offsets-37
> (kafka.coordinator.GroupMetadataMa
> nager)
> [2017-07-31 19:32:39,248] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions bseEquityAggFeedProd-4 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,249] INFO Truncating log bseEquityAggFeedProd-4 to
> offset 13082598. (kafka.log.Log)
> [2017-07-31 19:32:39,276] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([bseEquityAggFeedProd-4, initOffset 13082598 to
> broker BrokerEnd
> Point(2,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,277] INFO [ReplicaFetcherThread-0-2], Starting
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,294] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-17 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,294] INFO Truncating log __consumer_offsets-17 to
> offset 0. (kafka.log.Log)
> [2017-07-31 19:32:39,300] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([__consumer_offsets-17, initOffset 0 to broker
> BrokerEndPoint(2,
> 172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,304] ERROR [ReplicaFetcherThread-0-5], Error for
> partition [__consumer_offsets,9] to broker 5:org.apache.kafka.common.
> errors.NotLeaderForPartiti
> onException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,308] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [bseEquityAggFeedProd,4] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForParti
> tionException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,313] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [__consumer_offsets,17] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForPartit
> ionException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,318] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-9 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,319] INFO Truncating log __consumer_offsets-9 to
> offset 392. (kafka.log.Log)
> [2017-07-31 19:32:39,324] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([__consumer_offsets-9, initOffset 392 to broker
> BrokerEndPoint(2
> ,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,339] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [__consumer_offsets,9] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForPartiti
> onException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,986] INFO [GroupCoordinator 6]: Loading group
> metadata for bseEquityLegacyBod with generation 4 (kafka.coordinator.
> GroupCoordinator)
> [2017-07-31 19:32:39,991] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-37 in 759 milliseconds.
> (kafka.coordinator.Grou
> pMetadataManager)
>
> Consumer Logs:
>
> 17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Discovered
> coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group
> bseEquityLegacyBod.
> 17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Marking
> the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for
> group bseEquityLegacyBod
> 17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Discovered
> coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group
> bseEquityLegacyBod.
> 17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Marking
> the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for
> group bseEquityLegacyBod
>
> Thanks in advance.
>
> Regards,
> Mahesh Patade
>
>
> ________________________________
>
> DISCLAIMER AND PRIVILEGE NOTICE : This e-mail and any files transmitted
> with it contain confidential, copyright, proprietary and legally privileged
> information. It should not be used by anyone who is not the original
> intended recipient. Any use, distribution, copying or disclosure by any
> other person is strictly prohibited. If you receive this transmission in
> error, please notify the sender by reply email and then destroy the
> message. Opinions, conclusions and other information in this message that
> do not relate to official business of Network18 Media & Investments Ltd.,
> its subsidiaries, holding companies i.e. Network18 group shall be
> understood to be neither given nor endorsed by Network18 group. Internet
> communications cannot be guaranteed to be timely, Secure, error or
> virus-free. The sender does not accept liability for any errors or
> omissions.
>