You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghav <ra...@gmail.com> on 2018/02/06 20:24:27 UTC
log.retention.bytes not working as expected
Hi
While configuring a topic, we are specifying the retention bytes per topic
as follows. Our retention time in hours is 48.
*bin/kafka-topics.sh, --zookeeper zk-1:2181,zk-2:2181,zk-3:2181 --create
--topic AmazingTopic --replication-factor 2 --partitions 64 --config
retention.bytes=16106127360 --force*
According to Kafka documentation, when either of the condition is met, the
deletion of the log happens from the rear end.
Can anyone tell as to why retention.bytes is not working ?
Here is the relevant Kafka config:
# The minimum age of a log file to be eligible for deletion
*log.retention.hours=48*
# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
*log.retention.bytes=42949672960*
# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
*log.segment.bytes=1073741824*
# The interval at which log segments are checked to see if they can be
deleted according
# to the retention policies
*log.retention.check.interval.ms
<http://log.retention.check.interval.ms>=300000*
# By default the log cleaner is disabled and the log retention policy will
default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and
individual logs can then be marked for log compaction.
*log.cleaner.enable=true*
Thanks.
Re: log.retention.bytes not working as expected
Posted by Raghav <ra...@gmail.com>.
Linux. CentOS.
On Tue, Feb 6, 2018 at 12:26 PM, M. Manna <ma...@gmail.com> wrote:
> Is this Windows or Linux?
>
> On 6 Feb 2018 8:24 pm, "Raghav" <ra...@gmail.com> wrote:
>
> > Hi
> >
> > While configuring a topic, we are specifying the retention bytes per
> topic
> > as follows. Our retention time in hours is 48.
> >
> > *bin/kafka-topics.sh, --zookeeper zk-1:2181,zk-2:2181,zk-3:2181 --create
> > --topic AmazingTopic --replication-factor 2 --partitions 64 --config
> > retention.bytes=16106127360 --force*
> >
> > According to Kafka documentation, when either of the condition is met,
> the
> > deletion of the log happens from the rear end.
> >
> > Can anyone tell as to why retention.bytes is not working ?
> >
> > Here is the relevant Kafka config:
> >
> >
> > # The minimum age of a log file to be eligible for deletion
> > *log.retention.hours=48*
> >
> > # A size-based retention policy for logs. Segments are pruned from the
> log
> > as long as the remaining
> > # segments don't drop below log.retention.bytes.
> > *log.retention.bytes=42949672960*
> >
> > # The maximum size of a log segment file. When this size is reached a new
> > log segment will be created.
> > *log.segment.bytes=1073741824*
> >
> > # The interval at which log segments are checked to see if they can be
> > deleted according
> > # to the retention policies
> > *log.retention.check.interval.ms
> > <http://log.retention.check.interval.ms>=300000*
> >
> > # By default the log cleaner is disabled and the log retention policy
> will
> > default to just delete segments after their retention expires.
> > # If log.cleaner.enable=true is set the cleaner will be enabled and
> > individual logs can then be marked for log compaction.
> > *log.cleaner.enable=true*
> >
> > Thanks.
> >
>
--
Raghav
Re: log.retention.bytes not working as expected
Posted by "M. Manna" <ma...@gmail.com>.
Is this Windows or Linux?
On 6 Feb 2018 8:24 pm, "Raghav" <ra...@gmail.com> wrote:
> Hi
>
> While configuring a topic, we are specifying the retention bytes per topic
> as follows. Our retention time in hours is 48.
>
> *bin/kafka-topics.sh, --zookeeper zk-1:2181,zk-2:2181,zk-3:2181 --create
> --topic AmazingTopic --replication-factor 2 --partitions 64 --config
> retention.bytes=16106127360 --force*
>
> According to Kafka documentation, when either of the condition is met, the
> deletion of the log happens from the rear end.
>
> Can anyone tell as to why retention.bytes is not working ?
>
> Here is the relevant Kafka config:
>
>
> # The minimum age of a log file to be eligible for deletion
> *log.retention.hours=48*
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> *log.retention.bytes=42949672960*
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> *log.segment.bytes=1073741824*
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> *log.retention.check.interval.ms
> <http://log.retention.check.interval.ms>=300000*
>
> # By default the log cleaner is disabled and the log retention policy will
> default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> *log.cleaner.enable=true*
>
> Thanks.
>
Re: log.retention.bytes not working as expected
Posted by Raghav <ra...@gmail.com>.
By the way, is there a bug that was fixed in the later release.
https://issues.apache.org/jira/browse/KAFKA-6030
Can you please confirm ?
On Tue, Feb 6, 2018 at 1:38 PM, Ted Yu <yu...@gmail.com> wrote:
> The log cleaner abortion in the log file preceded log deletion.
>
> On Tue, Feb 6, 2018 at 1:36 PM, Raghav <ra...@gmail.com> wrote:
>
> > Ted
> >
> > Sorry, I did not understand your point here.
> >
> > On Tue, Feb 6, 2018 at 1:09 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. but is aborted.
> > >
> > > See the following in LogManager#asyncDelete():
> > >
> > > //We need to wait until there is no more cleaning task on the log
> > to
> > > be deleted before actually deleting it.
> > >
> > > if (cleaner != null && !isFuture) {
> > >
> > > cleaner.abortCleaning(topicPartition)
> > >
> > > FYI
> > >
> > > On Tue, Feb 6, 2018 at 12:56 PM, Raghav <ra...@gmail.com> wrote:
> > >
> > > > From the log-cleaner.log, I see the following. It seems like it
> resume
> > > but
> > > > is aborted. Not sure how to read this:
> > > >
> > > >
> > > > [2018-02-06 18:06:22,178] INFO Compaction for partition topic043-27
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,178] INFO The cleaning for partition topic043-27
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,182] INFO Compaction for partition topic043-51
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,190] INFO Compaction for partition topic043-52
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,192] INFO Compaction for partition topic043-45
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,198] INFO Compaction for partition topic043-20
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,204] INFO Compaction for partition topic043-63
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,209] INFO Compaction for partition topic043-44
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,214] INFO Compaction for partition topic043-38
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50
> > is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,219] INFO Compaction for partition topic043-50
> is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50
> > is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2
> is
> > > > aborted and paused (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,224] INFO Compaction for partition topic043-2 is
> > > > resumed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2
> is
> > > > aborted (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:54,643] INFO Shutting down the log cleaner.
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:54,643] INFO [kafka-log-cleaner-thread-0], Shutting
> > > down
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Shutdown
> > > > completed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Stopped
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:57,663] INFO Starting the log cleaner
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:06:57,665] INFO [kafka-log-cleaner-thread-0], Starting
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:07,187] INFO Shutting down the log cleaner.
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutting
> > > down
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Stopped
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutdown
> > > > completed (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:11,701] INFO Starting the log cleaner
> > > > (kafka.log.LogCleaner)
> > > > [2018-02-06 18:08:11,703] INFO [kafka-log-cleaner-thread-0], Starting
> > > > (kafka.log.LogCleaner)
> > > >
> > >
> >
> >
> >
> > --
> > Raghav
> >
>
--
Raghav
Re: log.retention.bytes not working as expected
Posted by Ted Yu <yu...@gmail.com>.
The log cleaner abortion in the log file preceded log deletion.
On Tue, Feb 6, 2018 at 1:36 PM, Raghav <ra...@gmail.com> wrote:
> Ted
>
> Sorry, I did not understand your point here.
>
> On Tue, Feb 6, 2018 at 1:09 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. but is aborted.
> >
> > See the following in LogManager#asyncDelete():
> >
> > //We need to wait until there is no more cleaning task on the log
> to
> > be deleted before actually deleting it.
> >
> > if (cleaner != null && !isFuture) {
> >
> > cleaner.abortCleaning(topicPartition)
> >
> > FYI
> >
> > On Tue, Feb 6, 2018 at 12:56 PM, Raghav <ra...@gmail.com> wrote:
> >
> > > From the log-cleaner.log, I see the following. It seems like it resume
> > but
> > > is aborted. Not sure how to read this:
> > >
> > >
> > > [2018-02-06 18:06:22,178] INFO Compaction for partition topic043-27 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,178] INFO The cleaning for partition topic043-27
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,182] INFO Compaction for partition topic043-51 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,190] INFO Compaction for partition topic043-52 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,192] INFO Compaction for partition topic043-45 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,198] INFO Compaction for partition topic043-20 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,204] INFO Compaction for partition topic043-63 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,209] INFO Compaction for partition topic043-44 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,214] INFO Compaction for partition topic043-38 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50
> is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,219] INFO Compaction for partition topic043-50 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50
> is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> > > aborted and paused (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,224] INFO Compaction for partition topic043-2 is
> > > resumed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> > > aborted (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:54,643] INFO Shutting down the log cleaner.
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:54,643] INFO [kafka-log-cleaner-thread-0], Shutting
> > down
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Shutdown
> > > completed (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Stopped
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:57,663] INFO Starting the log cleaner
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:06:57,665] INFO [kafka-log-cleaner-thread-0], Starting
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:07,187] INFO Shutting down the log cleaner.
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutting
> > down
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Stopped
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutdown
> > > completed (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:11,701] INFO Starting the log cleaner
> > > (kafka.log.LogCleaner)
> > > [2018-02-06 18:08:11,703] INFO [kafka-log-cleaner-thread-0], Starting
> > > (kafka.log.LogCleaner)
> > >
> >
>
>
>
> --
> Raghav
>
Re: log.retention.bytes not working as expected
Posted by Raghav <ra...@gmail.com>.
Ted
Sorry, I did not understand your point here.
On Tue, Feb 6, 2018 at 1:09 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. but is aborted.
>
> See the following in LogManager#asyncDelete():
>
> //We need to wait until there is no more cleaning task on the log to
> be deleted before actually deleting it.
>
> if (cleaner != null && !isFuture) {
>
> cleaner.abortCleaning(topicPartition)
>
> FYI
>
> On Tue, Feb 6, 2018 at 12:56 PM, Raghav <ra...@gmail.com> wrote:
>
> > From the log-cleaner.log, I see the following. It seems like it resume
> but
> > is aborted. Not sure how to read this:
> >
> >
> > [2018-02-06 18:06:22,178] INFO Compaction for partition topic043-27 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,178] INFO The cleaning for partition topic043-27 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,182] INFO Compaction for partition topic043-51 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,190] INFO Compaction for partition topic043-52 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,192] INFO Compaction for partition topic043-45 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,198] INFO Compaction for partition topic043-20 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,204] INFO Compaction for partition topic043-63 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,209] INFO Compaction for partition topic043-44 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,214] INFO Compaction for partition topic043-38 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,219] INFO Compaction for partition topic043-50 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> > aborted and paused (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,224] INFO Compaction for partition topic043-2 is
> > resumed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> > aborted (kafka.log.LogCleaner)
> > [2018-02-06 18:06:54,643] INFO Shutting down the log cleaner.
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:06:54,643] INFO [kafka-log-cleaner-thread-0], Shutting
> down
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Shutdown
> > completed (kafka.log.LogCleaner)
> > [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Stopped
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:06:57,663] INFO Starting the log cleaner
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:06:57,665] INFO [kafka-log-cleaner-thread-0], Starting
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:08:07,187] INFO Shutting down the log cleaner.
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutting
> down
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Stopped
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutdown
> > completed (kafka.log.LogCleaner)
> > [2018-02-06 18:08:11,701] INFO Starting the log cleaner
> > (kafka.log.LogCleaner)
> > [2018-02-06 18:08:11,703] INFO [kafka-log-cleaner-thread-0], Starting
> > (kafka.log.LogCleaner)
> >
>
--
Raghav
Re: log.retention.bytes not working as expected
Posted by Ted Yu <yu...@gmail.com>.
bq. but is aborted.
See the following in LogManager#asyncDelete():
//We need to wait until there is no more cleaning task on the log to
be deleted before actually deleting it.
if (cleaner != null && !isFuture) {
cleaner.abortCleaning(topicPartition)
FYI
On Tue, Feb 6, 2018 at 12:56 PM, Raghav <ra...@gmail.com> wrote:
> From the log-cleaner.log, I see the following. It seems like it resume but
> is aborted. Not sure how to read this:
>
>
> [2018-02-06 18:06:22,178] INFO Compaction for partition topic043-27 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,178] INFO The cleaning for partition topic043-27 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,182] INFO Compaction for partition topic043-51 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,190] INFO Compaction for partition topic043-52 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,192] INFO Compaction for partition topic043-45 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,198] INFO Compaction for partition topic043-20 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,204] INFO Compaction for partition topic043-63 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,209] INFO Compaction for partition topic043-44 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,214] INFO Compaction for partition topic043-38 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,219] INFO Compaction for partition topic043-50 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> aborted and paused (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,224] INFO Compaction for partition topic043-2 is
> resumed (kafka.log.LogCleaner)
> [2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
> aborted (kafka.log.LogCleaner)
> [2018-02-06 18:06:54,643] INFO Shutting down the log cleaner.
> (kafka.log.LogCleaner)
> [2018-02-06 18:06:54,643] INFO [kafka-log-cleaner-thread-0], Shutting down
> (kafka.log.LogCleaner)
> [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Shutdown
> completed (kafka.log.LogCleaner)
> [2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Stopped
> (kafka.log.LogCleaner)
> [2018-02-06 18:06:57,663] INFO Starting the log cleaner
> (kafka.log.LogCleaner)
> [2018-02-06 18:06:57,665] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
> [2018-02-06 18:08:07,187] INFO Shutting down the log cleaner.
> (kafka.log.LogCleaner)
> [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutting down
> (kafka.log.LogCleaner)
> [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Stopped
> (kafka.log.LogCleaner)
> [2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutdown
> completed (kafka.log.LogCleaner)
> [2018-02-06 18:08:11,701] INFO Starting the log cleaner
> (kafka.log.LogCleaner)
> [2018-02-06 18:08:11,703] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
>
Re: log.retention.bytes not working as expected
Posted by Raghav <ra...@gmail.com>.
From the log-cleaner.log, I see the following. It seems like it resume but
is aborted. Not sure how to read this:
[2018-02-06 18:06:22,178] INFO Compaction for partition topic043-27 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,178] INFO The cleaning for partition topic043-27 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,182] INFO Compaction for partition topic043-51 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,182] INFO The cleaning for partition topic043-51 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,190] INFO Compaction for partition topic043-52 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,190] INFO The cleaning for partition topic043-52 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,192] INFO Compaction for partition topic043-45 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,192] INFO The cleaning for partition topic043-45 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,198] INFO Compaction for partition topic043-20 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,198] INFO The cleaning for partition topic043-20 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,204] INFO Compaction for partition topic043-63 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,204] INFO The cleaning for partition topic043-63 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,209] INFO Compaction for partition topic043-44 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,209] INFO The cleaning for partition topic043-44 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,214] INFO Compaction for partition topic043-38 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,214] INFO The cleaning for partition topic043-38 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,219] INFO Compaction for partition topic043-50 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,219] INFO The cleaning for partition topic043-50 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
aborted and paused (kafka.log.LogCleaner)
[2018-02-06 18:06:22,224] INFO Compaction for partition topic043-2 is
resumed (kafka.log.LogCleaner)
[2018-02-06 18:06:22,224] INFO The cleaning for partition topic043-2 is
aborted (kafka.log.LogCleaner)
[2018-02-06 18:06:54,643] INFO Shutting down the log cleaner.
(kafka.log.LogCleaner)
[2018-02-06 18:06:54,643] INFO [kafka-log-cleaner-thread-0], Shutting down
(kafka.log.LogCleaner)
[2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Shutdown
completed (kafka.log.LogCleaner)
[2018-02-06 18:06:54,644] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
[2018-02-06 18:06:57,663] INFO Starting the log cleaner
(kafka.log.LogCleaner)
[2018-02-06 18:06:57,665] INFO [kafka-log-cleaner-thread-0], Starting
(kafka.log.LogCleaner)
[2018-02-06 18:08:07,187] INFO Shutting down the log cleaner.
(kafka.log.LogCleaner)
[2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutting down
(kafka.log.LogCleaner)
[2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
[2018-02-06 18:08:07,187] INFO [kafka-log-cleaner-thread-0], Shutdown
completed (kafka.log.LogCleaner)
[2018-02-06 18:08:11,701] INFO Starting the log cleaner
(kafka.log.LogCleaner)
[2018-02-06 18:08:11,703] INFO [kafka-log-cleaner-thread-0], Starting
(kafka.log.LogCleaner)
Re: log.retention.bytes not working as expected
Posted by "M. Manna" <ma...@gmail.com>.
Could you provide any broker/zk logs ? Zookeeper and Kafka logs a lot of
info during housekeeping ops such as log retention....there must be
something there..
On 6 Feb 2018 8:24 pm, "Raghav" <ra...@gmail.com> wrote:
> Hi
>
> While configuring a topic, we are specifying the retention bytes per topic
> as follows. Our retention time in hours is 48.
>
> *bin/kafka-topics.sh, --zookeeper zk-1:2181,zk-2:2181,zk-3:2181 --create
> --topic AmazingTopic --replication-factor 2 --partitions 64 --config
> retention.bytes=16106127360 --force*
>
> According to Kafka documentation, when either of the condition is met, the
> deletion of the log happens from the rear end.
>
> Can anyone tell as to why retention.bytes is not working ?
>
> Here is the relevant Kafka config:
>
>
> # The minimum age of a log file to be eligible for deletion
> *log.retention.hours=48*
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> *log.retention.bytes=42949672960*
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> *log.segment.bytes=1073741824*
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> *log.retention.check.interval.ms
> <http://log.retention.check.interval.ms>=300000*
>
> # By default the log cleaner is disabled and the log retention policy will
> default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> *log.cleaner.enable=true*
>
> Thanks.
>