Re: Getting error while consuming data from broker

["M. Manna" <ma...@gmail.com>]

Mahesh,

Thanks for sharing the info. Is having "Exactly" 8 brokers a "Must" for
you? because one of them is technically unnecessary since your cluster can
only tolerate 3 failures (even with 7 brokers).
Could you please try the following:

1) Stop the cluster.
2) Increase the number of renum.recovery.threads.per.data.dir from 1 to 5
(i mean something bigger than 1).
3) Retry creating a topic with both partitions and replication-factor set
to 8.

Let us know how it goes.

Regards,

On 31 July 2017 at 18:44, Mahesh Patade <Ma...@nw18.com> wrote:

> Hi All,
> We are having 8 broker kafka cluster configured in our setup and created a
> topic with 8 partitions & 3 replicas. While trying to consume data from one
> broker(id:6) we are getting below errors and increase in lag for active
> partition on that host. We even tried restarting, deleting logs and
> reinstalling broker but still no luck. This is happening only for one
> broker. If we shut down that broker service  everything works well.
> Kafka Server Logs:
> [2017-07-31 19:32:39,212] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-5 in 5 milliseconds.
> (kafka.coordinator.GroupMe
> tadataManager)
> [2017-07-31 19:32:39,217] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-13 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,218] INFO [Group Metadata Manager on Broker 6]:
> Loading offsets and group metadata from __consumer_offsets-13
> (kafka.coordinator.GroupMetadataMa
> nager)
> [2017-07-31 19:32:39,220] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-37 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,230] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-45 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-13 in 13 milliseconds.
> (kafka.coordinator.Group
> MetadataManager)
> [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]:
> Loading offsets and group metadata from __consumer_offsets-37
> (kafka.coordinator.GroupMetadataMa
> nager)
> [2017-07-31 19:32:39,248] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions bseEquityAggFeedProd-4 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,249] INFO Truncating log bseEquityAggFeedProd-4 to
> offset 13082598. (kafka.log.Log)
> [2017-07-31 19:32:39,276] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([bseEquityAggFeedProd-4, initOffset 13082598 to
> broker BrokerEnd
> Point(2,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,277] INFO [ReplicaFetcherThread-0-2], Starting
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,294] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-17 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,294] INFO Truncating log __consumer_offsets-17 to
> offset 0. (kafka.log.Log)
> [2017-07-31 19:32:39,300] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([__consumer_offsets-17, initOffset 0 to broker
> BrokerEndPoint(2,
> 172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,304] ERROR [ReplicaFetcherThread-0-5], Error for
> partition [__consumer_offsets,9] to broker 5:org.apache.kafka.common.
> errors.NotLeaderForPartiti
> onException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,308] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [bseEquityAggFeedProd,4] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForParti
> tionException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,313] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [__consumer_offsets,17] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForPartit
> ionException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,318] INFO [ReplicaFetcherManager on broker 6] Removed
> fetcher for partitions __consumer_offsets-9 (kafka.server.
> ReplicaFetcherManager)
> [2017-07-31 19:32:39,319] INFO Truncating log __consumer_offsets-9 to
> offset 392. (kafka.log.Log)
> [2017-07-31 19:32:39,324] INFO [ReplicaFetcherManager on broker 6] Added
> fetcher for partitions List([__consumer_offsets-9, initOffset 392 to broker
> BrokerEndPoint(2
> ,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-07-31 19:32:39,339] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [__consumer_offsets,9] to broker 2:org.apache.kafka.common.
> errors.NotLeaderForPartiti
> onException: This server is not the leader for that topic-partition.
> (kafka.server.ReplicaFetcherThread)
> [2017-07-31 19:32:39,986] INFO [GroupCoordinator 6]: Loading group
> metadata for bseEquityLegacyBod with generation 4 (kafka.coordinator.
> GroupCoordinator)
> [2017-07-31 19:32:39,991] INFO [Group Metadata Manager on Broker 6]:
> Finished loading offsets from __consumer_offsets-37 in 759 milliseconds.
> (kafka.coordinator.Grou
> pMetadataManager)
>
> Consumer Logs:
>
> 17/07/31/19:26:20.005 2-thread-3  INFO nals.AbstractCoordinator Discovered
> coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group
> bseEquityLegacyBod.
> 17/07/31/19:26:20.005 2-thread-3  INFO nals.AbstractCoordinator Marking
> the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for
> group bseEquityLegacyBod
> 17/07/31/19:26:20.106 2-thread-3  INFO nals.AbstractCoordinator Discovered
> coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group
> bseEquityLegacyBod.
> 17/07/31/19:26:20.106 2-thread-3  INFO nals.AbstractCoordinator Marking
> the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for
> group bseEquityLegacyBod
>
> Thanks in advance.
>
> Regards,
> Mahesh Patade
>
>
> ________________________________
>
> DISCLAIMER AND PRIVILEGE NOTICE : This e-mail and any files transmitted
> with it contain confidential, copyright, proprietary and legally privileged
> information. It should not be used by anyone who is not the original
> intended recipient. Any use, distribution, copying or disclosure by any
> other person is strictly prohibited. If you receive this transmission in
> error, please notify the sender by reply email and then destroy the
> message. Opinions, conclusions and other information in this message that
> do not relate to official business of Network18 Media & Investments Ltd.,
> its subsidiaries, holding companies i.e. Network18 group shall be
> understood to be neither given nor endorsed by Network18 group. Internet
> communications cannot be guaranteed to be timely, Secure, error or
> virus-free. The sender does not accept liability for any errors or
> omissions.
>