You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2018/07/30 09:31:01 UTC
Kafka sometime truncate new messages
I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to
0.11.0.3, I notice that Kafka sometimes truncate new message's topic. Here
is the log:
[2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-7. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-22. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-19. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-4. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-1. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-16. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-13. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-10. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,615] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-23. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-8. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-5. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-20. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-17. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-2. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-14. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
follower's leader epoch, leader replied with an unknown offset in
7798-11. High watermark 0 will be used for truncation.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,628] INFO Truncating 7798-20 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-17 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-14 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-11 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-8 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-2 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-5 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,628] INFO Truncating 7798-23 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,613] INFO Truncating 7798-1 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-22 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-16 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-19 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-13 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-10 to 0 has no effect
as the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-7 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,637] INFO Truncating 7798-4 to 0 has no effect as
the largest offset in the log is -1. (kafka.log.Log)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,8] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,2] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,23] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,17] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,5] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,14] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,11] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,709] ERROR [ReplicaFetcherThread-0-102]: Error
for partition [7798,20] to broker
102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,7] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,13] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,22] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,4] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,10] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,19] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error
for partition [7798,1] to broker
101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
t seems at the first Kafka save the message and after that, it started to
truncate messages, right away! I guess it's because of resolving this bug
<https://issues.apache.org/jira/browse/KAFKA-6857> in Kafka version 11.0.3.
This scenario happened even for __consumer_offset topic. Is there any way
to solve the problem?
Re: Kafka sometime truncate new messages
Posted by Soheil Pourbafrani <so...@gmail.com>.
I didn't test that. All other software is compatible with Kafka 0.11 and we
can not change it easily.
On Mon, Jul 30, 2018 at 5:07 PM, M. Manna <ma...@gmail.com> wrote:
> The release came out earlier today 2.0.0 - Does this not work for you if
> you upgrade to that?
>
> On 30 July 2018 at 13:02, Soheil Pourbafrani <so...@gmail.com>
> wrote:
>
> > In addition, I should say this behavior is for when topics are created
> > automatically.
> >
> > On Mon, Jul 30, 2018 at 2:01 PM, Soheil Pourbafrani <
> soheil.ir08@gmail.com
> > >
> > wrote:
> >
> > > I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to
> > > 0.11.0.3, I notice that Kafka sometimes truncate new message's topic.
> > Here
> > > is the log:
> > >
> > > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-7.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-22.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-19.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-4.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-1.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-16.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-13.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-10.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,615] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-23.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-8.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-5.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-20.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-17.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in 7798-2.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-14.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> > follower's leader epoch, leader replied with an unknown offset in
> 7798-11.
> > High watermark 0 will be used for truncation. (kafka.server.
> > ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-20 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-17 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-14 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-11 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-8 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-2 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-5 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,628] INFO Truncating 7798-23 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,613] INFO Truncating 7798-1 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-22 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-16 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-19 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-13 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-10 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-7 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,637] INFO Truncating 7798-4 to 0 has no effect as
> > the largest offset in the log is -1. (kafka.log.Log)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,8] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,2] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,23] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,17] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,5] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,14] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,11] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,709] ERROR [ReplicaFetcherThread-0-102]: Error for
> > partition [7798,20] to broker 102:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,7] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,13] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,22] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,4] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,10] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,19] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> > partition [7798,1] to broker 101:org.apache.kafka.common.errors.
> > UnknownTopicOrPartitionException: This server does not host this
> > topic-partition. (kafka.server.ReplicaFetcherThread)
> > >
> > > t seems at the first Kafka save the message and after that, it started
> to
> > > truncate messages, right away! I guess it's because of resolving this
> bug
> > > <https://issues.apache.org/jira/browse/KAFKA-6857> in Kafka version
> > > 11.0.3. This scenario happened even for __consumer_offset topic. Is
> there
> > > any way to solve the problem?
> > >
> > >
> > >
> >
>
Re: Kafka sometime truncate new messages
Posted by "M. Manna" <ma...@gmail.com>.
The release came out earlier today 2.0.0 - Does this not work for you if
you upgrade to that?
On 30 July 2018 at 13:02, Soheil Pourbafrani <so...@gmail.com> wrote:
> In addition, I should say this behavior is for when topics are created
> automatically.
>
> On Mon, Jul 30, 2018 at 2:01 PM, Soheil Pourbafrani <soheil.ir08@gmail.com
> >
> wrote:
>
> > I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to
> > 0.11.0.3, I notice that Kafka sometimes truncate new message's topic.
> Here
> > is the log:
> >
> > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-7.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-22.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-19.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-4.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-1.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-16.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-13.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-10.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,615] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-23.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-8.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-5.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-20.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-17.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-2.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-14.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on
> follower's leader epoch, leader replied with an unknown offset in 7798-11.
> High watermark 0 will be used for truncation. (kafka.server.
> ReplicaFetcherThread)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-20 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-17 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-14 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-11 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-8 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-2 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-5 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,628] INFO Truncating 7798-23 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,613] INFO Truncating 7798-1 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-22 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-16 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-19 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-13 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-10 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-7 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,637] INFO Truncating 7798-4 to 0 has no effect as
> the largest offset in the log is -1. (kafka.log.Log)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,8] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,2] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,23] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,17] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,5] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,14] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,11] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,709] ERROR [ReplicaFetcherThread-0-102]: Error for
> partition [7798,20] to broker 102:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,7] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,13] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,22] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,4] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,10] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,19] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> > [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for
> partition [7798,1] to broker 101:org.apache.kafka.common.errors.
> UnknownTopicOrPartitionException: This server does not host this
> topic-partition. (kafka.server.ReplicaFetcherThread)
> >
> > t seems at the first Kafka save the message and after that, it started to
> > truncate messages, right away! I guess it's because of resolving this bug
> > <https://issues.apache.org/jira/browse/KAFKA-6857> in Kafka version
> > 11.0.3. This scenario happened even for __consumer_offset topic. Is there
> > any way to solve the problem?
> >
> >
> >
>
Re: Kafka sometime truncate new messages
Posted by Soheil Pourbafrani <so...@gmail.com>.
In addition, I should say this behavior is for when topics are created
automatically.
On Mon, Jul 30, 2018 at 2:01 PM, Soheil Pourbafrani <so...@gmail.com>
wrote:
> I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to
> 0.11.0.3, I notice that Kafka sometimes truncate new message's topic. Here
> is the log:
>
> [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-7. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-22. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-19. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-4. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-1. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-16. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-13. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,613] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-10. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,615] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-23. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-8. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-5. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-20. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-17. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-2. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-14. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,627] WARN [ReplicaFetcherThread-0-102]: Based on follower's leader epoch, leader replied with an unknown offset in 7798-11. High watermark 0 will be used for truncation. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-20 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-17 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-14 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-11 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-8 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-2 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-5 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,628] INFO Truncating 7798-23 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,613] INFO Truncating 7798-1 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-22 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-16 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-19 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-13 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-10 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-7 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,637] INFO Truncating 7798-4 to 0 has no effect as the largest offset in the log is -1. (kafka.log.Log)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,8] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,2] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,23] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,17] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,5] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,14] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,708] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,11] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,709] ERROR [ReplicaFetcherThread-0-102]: Error for partition [7798,20] to broker 102:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,7] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,13] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,714] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,22] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,4] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,10] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,19] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2018-07-30 09:19:39,715] ERROR [ReplicaFetcherThread-0-101]: Error for partition [7798,1] to broker 101:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
>
> t seems at the first Kafka save the message and after that, it started to
> truncate messages, right away! I guess it's because of resolving this bug
> <https://issues.apache.org/jira/browse/KAFKA-6857> in Kafka version
> 11.0.3. This scenario happened even for __consumer_offset topic. Is there
> any way to solve the problem?
>
>
>