You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nicolas Carlot <ni...@chronopost.fr.INVALID> on 2022/12/22 15:30:00 UTC
log.cleaner.io.max.bytes.per.second
Hello users,
I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to
limit disk usage when the log cleaner runs, but it doesn't seem to work as
expected.
When looking at the logs, I can see it still uses far more than configured.
This is the output with log.cleaner.io.max.bytes.per.second=1024
[2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
section = [13509895330, 13510879425])
100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total time)
Buffer utilization: 0,1%
Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total time)
Start size: 100,2 MB (988642 messages)
End size: 0,5 MB (4547 messages)
99,5% size reduction (99,5% fewer messages)
I did update the value trough kafak-config this way:
kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
--command-config ~/server.admin.properties --entity-default --alter
--add-config log.cleaner.io.max.bytes.per.second=1024
Which shows up correctly in broker config:
log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io.max.bytes.per.second=1024,
DEFAULT_CONFIG:log.cleaner.io.max.bytes.per.second=1.7976931348623157E308}
Any idea ? Something I'm missing ?
--
[image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
[image: img]
Re: log.cleaner.io.max.bytes.per.second
Posted by Luke Chen <sh...@gmail.com>.
Hi Nicolas,
> Any Idea some may have to limit log compaction disk usage ? That's a
pretty
big issue we're having, compaction of a few topics is using far more IO
than everything else on the broker...
I don't know any other better solution except make the
`log.cleaner.io.max.bytes.per.second` lower than you expected, maybe 512?
As I explained earlier, based on how throttler works currently, there will
be a 300ms time slot of full speed read/write.
And it will make the final IO speed higher than you expected.
However, Apache Kafka is an open source project, so, welcome to make a
contribution to make it better!
Let me know if you're interested.
Thank you.
Luke
On Wed, Dec 28, 2022 at 6:25 PM Nicolas Carlot
<ni...@chronopost.fr.invalid> wrote:
> Okay, thx for the explaination Luc.
> When looking for logs on a bigger topic, I can see the IO limitation kinda
> works.
> I have like15MB/s when having configured
> log.cleaner.io.max.bytes.per.second=1024.0
> I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending
> with 15Mb/s, seems a bit akward.
> Any Idea some may have to limit log compaction disk usage ? That's a pretty
> big issue we're having, compaction of a few topics is using far more IO
> than everything else on the broker...
>
> Log cleaner thread 0 cleaned log
>
> kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8
> (dirty section = [34819497, 44148502])
> 1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec).
> Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total
> time)
> Buffer utilization: 9,2%
> Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of total
> time)
> Start size: 1 372,1 MB (10 329 108 messages)
> End size: 230,0 MB (1 000 105 messages)
> 83,2% size reduction (90,3% fewer messages)
>
> Le mer. 28 déc. 2022 à 09:41, Luke Chen <sh...@gmail.com> a écrit :
>
> > Hi Nicolas,
> >
> > The throttler in log cleaner is just a simple throttler to control the IO
> > speed by periodically (every 300ms) check the IO rate so far, and sleep
> > some time to slow it down.
> > That is, in your case, it could be:
> > log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
> > to reach desired rate) -> run 300ms with full speed -> done
> >
> > So, when printing the log, you will see the rate is higher than you
> > expected.
> > At least I ran in kafka v3.3.1 and it works well.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
> > <ni...@chronopost.fr.invalid> wrote:
> >
> > > Hello users,
> > >
> > > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration
> > to
> > > limit disk usage when the log cleaner runs, but it doesn't seem to work
> > as
> > > expected.
> > > When looking at the logs, I can see it still uses far more than
> > configured.
> > >
> > > This is the output with log.cleaner.io.max.bytes.per.second=1024
> > >
> > > [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> > > Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> > > section = [13509895330, 13510879425])
> > > 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> > > Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total
> > time)
> > > Buffer utilization: 0,1%
> > > Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total
> > time)
> > > Start size: 100,2 MB (988642 messages)
> > > End size: 0,5 MB (4547 messages)
> > > 99,5% size reduction (99,5% fewer messages)
> > >
> > > I did update the value trough kafak-config this way:
> > >
> > > kafka-configs.sh --bootstrap-server localhost:59092 --entity-type
> brokers
> > > --command-config ~/server.admin.properties --entity-default --alter
> > > --add-config log.cleaner.io.max.bytes.per.second=1024
> > >
> > > Which shows up correctly in broker config:
> > > log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> > > synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> > > .max.bytes.per.second=1024,
> > > DEFAULT_CONFIG:log.cleaner.io
> > .max.bytes.per.second=1.7976931348623157E308}
> > >
> > > Any idea ? Something I'm missing ?
> > >
> > > --
> > > [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > >
> > >
> > > *Nicolas Carlot*
> > > *Lead dev*Direction des Systèmes d'Information
> > >
> > >
> > > 3 boulevard Romain Rolland
> > > 75014 Paris
> > > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > > [image:
> > > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > > img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> > >
> > > [image: img]
> > >
> > > [image: img]
> > >
> >
>
>
> --
> [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
>
> [image: img]
>
Re: log.cleaner.io.max.bytes.per.second
Posted by Nicolas Carlot <ni...@chronopost.fr.INVALID>.
Okay, thx for the explaination Luc.
When looking for logs on a bigger topic, I can see the IO limitation kinda
works.
I have like15MB/s when having configured
log.cleaner.io.max.bytes.per.second=1024.0
I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending
with 15Mb/s, seems a bit akward.
Any Idea some may have to limit log compaction disk usage ? That's a pretty
big issue we're having, compaction of a few topics is using far more IO
than everything else on the broker...
Log cleaner thread 0 cleaned log
kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8
(dirty section = [34819497, 44148502])
1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec).
Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total
time)
Buffer utilization: 9,2%
Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of total
time)
Start size: 1 372,1 MB (10 329 108 messages)
End size: 230,0 MB (1 000 105 messages)
83,2% size reduction (90,3% fewer messages)
Le mer. 28 déc. 2022 à 09:41, Luke Chen <sh...@gmail.com> a écrit :
> Hi Nicolas,
>
> The throttler in log cleaner is just a simple throttler to control the IO
> speed by periodically (every 300ms) check the IO rate so far, and sleep
> some time to slow it down.
> That is, in your case, it could be:
> log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
> to reach desired rate) -> run 300ms with full speed -> done
>
> So, when printing the log, you will see the rate is higher than you
> expected.
> At least I ran in kafka v3.3.1 and it works well.
>
> Thank you.
> Luke
>
> On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
> <ni...@chronopost.fr.invalid> wrote:
>
> > Hello users,
> >
> > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration
> to
> > limit disk usage when the log cleaner runs, but it doesn't seem to work
> as
> > expected.
> > When looking at the logs, I can see it still uses far more than
> configured.
> >
> > This is the output with log.cleaner.io.max.bytes.per.second=1024
> >
> > [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> > Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> > section = [13509895330, 13510879425])
> > 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> > Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total
> time)
> > Buffer utilization: 0,1%
> > Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total
> time)
> > Start size: 100,2 MB (988642 messages)
> > End size: 0,5 MB (4547 messages)
> > 99,5% size reduction (99,5% fewer messages)
> >
> > I did update the value trough kafak-config this way:
> >
> > kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
> > --command-config ~/server.admin.properties --entity-default --alter
> > --add-config log.cleaner.io.max.bytes.per.second=1024
> >
> > Which shows up correctly in broker config:
> > log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> > synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> > .max.bytes.per.second=1024,
> > DEFAULT_CONFIG:log.cleaner.io
> .max.bytes.per.second=1.7976931348623157E308}
> >
> > Any idea ? Something I'm missing ?
> >
> > --
> > [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
> >
> >
> > *Nicolas Carlot*
> > *Lead dev*Direction des Systèmes d'Information
> >
> >
> > 3 boulevard Romain Rolland
> > 75014 Paris
> > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > [image:
> > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> >
> > [image: img]
> >
> > [image: img]
> >
>
--
[image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
[image: img]
Re: log.cleaner.io.max.bytes.per.second
Posted by Luke Chen <sh...@gmail.com>.
Hi Nicolas,
The throttler in log cleaner is just a simple throttler to control the IO
speed by periodically (every 300ms) check the IO rate so far, and sleep
some time to slow it down.
That is, in your case, it could be:
log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
to reach desired rate) -> run 300ms with full speed -> done
So, when printing the log, you will see the rate is higher than you
expected.
At least I ran in kafka v3.3.1 and it works well.
Thank you.
Luke
On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
<ni...@chronopost.fr.invalid> wrote:
> Hello users,
>
> I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to
> limit disk usage when the log cleaner runs, but it doesn't seem to work as
> expected.
> When looking at the logs, I can see it still uses far more than configured.
>
> This is the output with log.cleaner.io.max.bytes.per.second=1024
>
> [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> section = [13509895330, 13510879425])
> 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total time)
> Buffer utilization: 0,1%
> Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total time)
> Start size: 100,2 MB (988642 messages)
> End size: 0,5 MB (4547 messages)
> 99,5% size reduction (99,5% fewer messages)
>
> I did update the value trough kafak-config this way:
>
> kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
> --command-config ~/server.admin.properties --entity-default --alter
> --add-config log.cleaner.io.max.bytes.per.second=1024
>
> Which shows up correctly in broker config:
> log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> .max.bytes.per.second=1024,
> DEFAULT_CONFIG:log.cleaner.io.max.bytes.per.second=1.7976931348623157E308}
>
> Any idea ? Something I'm missing ?
>
> --
> [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
>
> [image: img]
>