You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Parth Gandhi <pa...@excellenceinfonet.com> on 2018/11/30 12:46:53 UTC
Setup GC log rotate policy for kafka broker servers
Team,
I took over kafka from a colleague and found that the GC log has been setup
without setting up the log rotate policy. Now the GC log size has reached
to 4 GB and we need to move this file and also set up a log rotate policy.
Can this be achieved without restarting the kafak service? We have a kafka
cluster with 3 broker and will have to set this up on all three servers. I
am new to Java and any help on this is highly appreciated.
Thanks,
Parth Gandhi
DevOps
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
Re: Setup GC log rotate policy for kafka broker servers
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello Parth,
I would recommend avoiding rapid restarts, it means, restarting the Brokers
immediately one after the other, also (in my opinion, avoid restarting the
Brokers all at once).
According to my experience and my opinion, it is better to restart one by
one, but wait until the restarted Broker catch-up/sync with the rest of the
Brokers and then continue with another, like this until you restart all of
them.
Hope this helps,
Cheers!
--
Jonathan
On Thu, Dec 6, 2018 at 10:58 AM Parth Gandhi <
parth.gandhi@excellenceinfonet.com> wrote:
> Thanks, Mani. IT seems the jmx code for GC log rotate has not been added to
> our run-class.sh in any of the server in the cluster. Also are there any
> best practice to restart the kafka service on broker in the cluster?
> Or we can just restart the service on all the broker server one by one and
> the rebalancing will occur automatically?
>
> Thanks,
> Parth Gandhi
> DevOps
>
>
> On Thu, Dec 6, 2018 at 4:56 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Parth,
> >
> > We need to pass JVM system properties to rotate gc logs. These props
> added
> > in Kafka 0.11.release.
> > For releases before 0.11, we need to add these JVM props. Broker restart
> is
> > required for changes to take effect.
> >
> > https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290
> > <https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290>
> >
> >
> >
> > On Fri, Nov 30, 2018 at 6:17 PM Parth Gandhi <
> > parth.gandhi@excellenceinfonet.com> wrote:
> >
> > > Team,
> > >
> > > I took over kafka from a colleague and found that the GC log has been
> > setup
> > > without setting up the log rotate policy. Now the GC log size has
> reached
> > > to 4 GB and we need to move this file and also set up a log rotate
> > policy.
> > > Can this be achieved without restarting the kafak service? We have a
> > kafka
> > > cluster with 3 broker and will have to set this up on all three
> servers.
> > I
> > > am new to Java and any help on this is highly appreciated.
> > >
> > > Thanks,
> > > Parth Gandhi
> > > DevOps
> > >
> > > Disclaimer
> > >
> > > The information contained in this communication from the sender is
> > > confidential. It is intended solely for use by the recipient and others
> > > authorized to receive it. If you are not the recipient, you are hereby
> > > notified that any disclosure, copying, distribution or taking action in
> > > relation of the contents of this information is strictly prohibited and
> > may
> > > be unlawful.
> > >
> > > This email has been scanned for viruses and malware, and may have been
> > > automatically archived by Mimecast Ltd, an innovator in Software as a
> > > Service (SaaS) for business. Providing a safer and more useful place
> for
> > > your human generated data. Specializing in; Security, archiving and
> > > compliance. To find out more visit the Mimecast website.
> > >
> >
>
> Disclaimer
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast Ltd, an innovator in Software as a
> Service (SaaS) for business. Providing a safer and more useful place for
> your human generated data. Specializing in; Security, archiving and
> compliance. To find out more visit the Mimecast website.
>
--
Santilli Jonathan
Re: Setup GC log rotate policy for kafka broker servers
Posted by Parth Gandhi <pa...@excellenceinfonet.com>.
Thanks, Mani. IT seems the jmx code for GC log rotate has not been added to
our run-class.sh in any of the server in the cluster. Also are there any
best practice to restart the kafka service on broker in the cluster?
Or we can just restart the service on all the broker server one by one and
the rebalancing will occur automatically?
Thanks,
Parth Gandhi
DevOps
On Thu, Dec 6, 2018 at 4:56 AM Manikumar <ma...@gmail.com> wrote:
> Hi Parth,
>
> We need to pass JVM system properties to rotate gc logs. These props added
> in Kafka 0.11.release.
> For releases before 0.11, we need to add these JVM props. Broker restart is
> required for changes to take effect.
>
> https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290
> <https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290>
>
>
>
> On Fri, Nov 30, 2018 at 6:17 PM Parth Gandhi <
> parth.gandhi@excellenceinfonet.com> wrote:
>
> > Team,
> >
> > I took over kafka from a colleague and found that the GC log has been
> setup
> > without setting up the log rotate policy. Now the GC log size has reached
> > to 4 GB and we need to move this file and also set up a log rotate
> policy.
> > Can this be achieved without restarting the kafak service? We have a
> kafka
> > cluster with 3 broker and will have to set this up on all three servers.
> I
> > am new to Java and any help on this is highly appreciated.
> >
> > Thanks,
> > Parth Gandhi
> > DevOps
> >
> > Disclaimer
> >
> > The information contained in this communication from the sender is
> > confidential. It is intended solely for use by the recipient and others
> > authorized to receive it. If you are not the recipient, you are hereby
> > notified that any disclosure, copying, distribution or taking action in
> > relation of the contents of this information is strictly prohibited and
> may
> > be unlawful.
> >
> > This email has been scanned for viruses and malware, and may have been
> > automatically archived by Mimecast Ltd, an innovator in Software as a
> > Service (SaaS) for business. Providing a safer and more useful place for
> > your human generated data. Specializing in; Security, archiving and
> > compliance. To find out more visit the Mimecast website.
> >
>
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
Re: Setup GC log rotate policy for kafka broker servers
Posted by Manikumar <ma...@gmail.com>.
Hi Parth,
We need to pass JVM system properties to rotate gc logs. These props added
in Kafka 0.11.release.
For releases before 0.11, we need to add these JVM props. Broker restart is
required for changes to take effect.
https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290
On Fri, Nov 30, 2018 at 6:17 PM Parth Gandhi <
parth.gandhi@excellenceinfonet.com> wrote:
> Team,
>
> I took over kafka from a colleague and found that the GC log has been setup
> without setting up the log rotate policy. Now the GC log size has reached
> to 4 GB and we need to move this file and also set up a log rotate policy.
> Can this be achieved without restarting the kafak service? We have a kafka
> cluster with 3 broker and will have to set this up on all three servers. I
> am new to Java and any help on this is highly appreciated.
>
> Thanks,
> Parth Gandhi
> DevOps
>
> Disclaimer
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast Ltd, an innovator in Software as a
> Service (SaaS) for business. Providing a safer and more useful place for
> your human generated data. Specializing in; Security, archiving and
> compliance. To find out more visit the Mimecast website.
>