You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vinod KC <VK...@hortonworks.com> on 2017/07/31 02:51:06 UTC

Kafka broker restart is taking more than 1 hr to complete

 Hi ,

Our business use case requires  to keep all kafka logs in topics for ever and  based on suggestion provided in http://grokbase.com/t/kafka/users/152ws9gbvx/unlimited-log-retention  ,we have set  log.retention.hours=2147483647"  (245,146 years)

We have around 45 TB data in 3 nodes kafka 0.8.2.2 cluster, but now each kafka broker is taking more than 1 hour to restart, do you think it is because of high data retention value?

Please refer the below time difference from shutdown to log loading completion

>cat /kafka-2.out | grep "Controlled shutdown succeeded" 
 [2017-06-12 05:18:28,190] INFO [Kafka Server 1002], Controlled shutdown succeeded (kafka.server.KafkaServer) 
> cat /kafka-2.out | grep "INFO starting (kafka.server.KafkaServer)" 
[2017-06-12 05:23:17,691] INFO starting (kafka.server.KafkaServer)
>cat /kafka-2.out | grep "INFO Logs loading complete. (kafka.log.LogManager)" 
 [2017-06-12 06:29:28,171] INFO Logs loading complete. (kafka.log.LogManager)

Thanks
Vinod

Re: Kafka broker restart is taking more than 1 hr to complete

Posted by Manikumar <ma...@gmail.com>.
Log loading is taking time.  By default, Kafka uses one thread per data.dir
to load the logs.
You can try by increasing  "num.recovery.threads.per.data.dir" broker
config property.


On Mon, Jul 31, 2017 at 8:21 AM, Vinod KC <VK...@hortonworks.com> wrote:

>  Hi ,
>
> Our business use case requires  to keep all kafka logs in topics for ever
> and  based on suggestion provided in http://grokbase.com/t/kafka/
> users/152ws9gbvx/unlimited-log-retention  ,we have set
> log.retention.hours=2147483647"  (245,146 years)
>
> We have around 45 TB data in 3 nodes kafka 0.8.2.2 cluster, but now each
> kafka broker is taking more than 1 hour to restart, do you think it is
> because of high data retention value?
>
> Please refer the below time difference from shutdown to log loading
> completion
>
> >cat /kafka-2.out | grep "Controlled shutdown succeeded"
>  [2017-06-12 05:18:28,190] INFO [Kafka Server 1002], Controlled shutdown
> succeeded (kafka.server.KafkaServer)
> > cat /kafka-2.out | grep "INFO starting (kafka.server.KafkaServer)"
> [2017-06-12 05:23:17,691] INFO starting (kafka.server.KafkaServer)
> >cat /kafka-2.out | grep "INFO Logs loading complete.
> (kafka.log.LogManager)"
>  [2017-06-12 06:29:28,171] INFO Logs loading complete.
> (kafka.log.LogManager)
>
> Thanks
> Vinod
>
>