You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Brebner, Paul" <Pa...@netapp.com.INVALID> on 2024/11/01 22:50:07 UTC
Re: Q: Does Kafka log level affect performance and latency
Hi Om, I asked some of our techops people about this, and their general advice is that increasing the log level from the default (INFO?) is likely to increase the I/O (the amount depending on a variety of factors including the cluster traffic etc) – and my take on this is that assuming there is sufficient I/O headroom it shouldn’t impact the cluster throughput or latency. However, in previous performance benchmarking I haven’t tried changing log levels so I would suggest you do some more detailed benchmarking with realistic workloads and increased log levels, good luck, Paul Brebner
From: Om Shree <om...@gmail.com>
Date: Friday, 1 November 2024 at 2:47 am
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Q: Does Kafka log level affect performance and latency
[You don't often get email from om22shree@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
Hello community,
We use Kafka extensively at our organisation.
Our SLAs are strict and require:-
— throughputs north of 1000TPS and
— 60ms latency per transaction
We run:-
— 6 brokers and 6 zookeepers
— brokers and zookeepers are hosted on ec2 instances with sufficient iops,
throughput and network bandwidth to meet our requirements
We were under the impression that using log levels (on brokers) such as
INFO or DEBUG would produce too many server logs and have an adverse impact
on performance.
Can someone with experience please confirm that log levels won’t have an
adverse effect on latency and throughput of these clusters or vice-versa,
given that we come up with a strategy to clean out these logs on broker
servers after a defined unit of time ?
Does generation of system logs have potential to impact disk or compute
iops and slow us down ?
Thanks!
Re: Q: Does Kafka log level affect performance and latency
Posted by Artem Timchenko <ar...@bolt.eu.INVALID>.
DEBUG level for some classes could have dramatic performance impact.
I tested DEBUG level for *kafka.authorizer.logger *for audit purposes, like
a naive approach to have audit logs about actions taken by engineers when
working with cluster directly. I don't have exact data right now, since it
was around 2 years ago, but I remember we managed to kill the cluster
completely by this change and had to rollback since it caused quite a wide
incident.
On Sat, Nov 2, 2024 at 12:53 AM Brebner, Paul
<Pa...@netapp.com.invalid> wrote:
> And the flip side, if the cluster is heavily loaded then increasing log
> levels is likely to have a detectable impact on performance! Paul
>
> From: Brebner, Paul <Pa...@netapp.com>
> Date: Saturday, 2 November 2024 at 9:50 am
> To: users@kafka.apache.org <us...@kafka.apache.org>, om22shree@gmail.com <
> om22shree@gmail.com>
> Subject: Re: Q: Does Kafka log level affect performance and latency
> Hi Om, I asked some of our techops people about this, and their general
> advice is that increasing the log level from the default (INFO?) is likely
> to increase the I/O (the amount depending on a variety of factors including
> the cluster traffic etc) – and my take on this is that assuming there is
> sufficient I/O headroom it shouldn’t impact the cluster throughput or
> latency. However, in previous performance benchmarking I haven’t tried
> changing log levels so I would suggest you do some more detailed
> benchmarking with realistic workloads and increased log levels, good luck,
> Paul Brebner
>
> From: Om Shree <om...@gmail.com>
> Date: Friday, 1 November 2024 at 2:47 am
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Subject: Q: Does Kafka log level affect performance and latency
> [You don't often get email from om22shree@gmail.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>
>
>
>
> Hello community,
>
> We use Kafka extensively at our organisation.
> Our SLAs are strict and require:-
> — throughputs north of 1000TPS and
> — 60ms latency per transaction
>
> We run:-
> — 6 brokers and 6 zookeepers
> — brokers and zookeepers are hosted on ec2 instances with sufficient iops,
> throughput and network bandwidth to meet our requirements
>
> We were under the impression that using log levels (on brokers) such as
> INFO or DEBUG would produce too many server logs and have an adverse impact
> on performance.
>
> Can someone with experience please confirm that log levels won’t have an
> adverse effect on latency and throughput of these clusters or vice-versa,
> given that we come up with a strategy to clean out these logs on broker
> servers after a defined unit of time ?
> Does generation of system logs have potential to impact disk or compute
> iops and slow us down ?
>
> Thanks!
>
Re: Q: Does Kafka log level affect performance and latency
Posted by "Brebner, Paul" <Pa...@netapp.com.INVALID>.
And the flip side, if the cluster is heavily loaded then increasing log levels is likely to have a detectable impact on performance! Paul
From: Brebner, Paul <Pa...@netapp.com>
Date: Saturday, 2 November 2024 at 9:50 am
To: users@kafka.apache.org <us...@kafka.apache.org>, om22shree@gmail.com <om...@gmail.com>
Subject: Re: Q: Does Kafka log level affect performance and latency
Hi Om, I asked some of our techops people about this, and their general advice is that increasing the log level from the default (INFO?) is likely to increase the I/O (the amount depending on a variety of factors including the cluster traffic etc) – and my take on this is that assuming there is sufficient I/O headroom it shouldn’t impact the cluster throughput or latency. However, in previous performance benchmarking I haven’t tried changing log levels so I would suggest you do some more detailed benchmarking with realistic workloads and increased log levels, good luck, Paul Brebner
From: Om Shree <om...@gmail.com>
Date: Friday, 1 November 2024 at 2:47 am
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Q: Does Kafka log level affect performance and latency
[You don't often get email from om22shree@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
Hello community,
We use Kafka extensively at our organisation.
Our SLAs are strict and require:-
— throughputs north of 1000TPS and
— 60ms latency per transaction
We run:-
— 6 brokers and 6 zookeepers
— brokers and zookeepers are hosted on ec2 instances with sufficient iops,
throughput and network bandwidth to meet our requirements
We were under the impression that using log levels (on brokers) such as
INFO or DEBUG would produce too many server logs and have an adverse impact
on performance.
Can someone with experience please confirm that log levels won’t have an
adverse effect on latency and throughput of these clusters or vice-versa,
given that we come up with a strategy to clean out these logs on broker
servers after a defined unit of time ?
Does generation of system logs have potential to impact disk or compute
iops and slow us down ?
Thanks!