You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Aniket Bhatnagar <an...@gmail.com> on 2013/10/04 09:45:09 UTC
Kafka stops accepting messages for existing topics after a hostname
change and restart
Because Kafka was detecting localhost.domain as hostname, I commented out
the line "127.0.0.1 localhost.localdomain localhost" and added
"127.0.0.1 ip-10-0-1-20.localdomain" in etc/hosts. When I restart Kafka
(issues kill -15 pid), writes to existing topics are failing and I see
several of the following errors:
[2013-10-04 03:43:23,565] WARN [KafkaApi-1] Produce request with
correlation id 12868451 from client on partition
[consumertracking.gnomes.4002,5] failed due to Partition
[consumertracking.gnomes.4002,5] doesn't exist on 1 (kafka.server.KafkaApis)
I have checked the kafka data directory and all files are intact. What
could have caused this issue and how can it be fixed?
Re: Kafka stops accepting messages for existing topics after a
hostname change and restart
Posted by Neha Narkhede <ne...@gmail.com>.
Controlled shutdown is described here -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown
On Fri, Oct 4, 2013 at 10:18 AM, Aniket Bhatnagar <
aniket.bhatnagar@gmail.com> wrote:
> I did restart broker very quickly. I saw similar errors for about 5 mins
> and that's when I decided to shutdown all kafka brokers and start them one
> by one. That seems to have enabled writes in kafka instantly after brokers
> were back up.
>
> How do I do a controlled shutdown? The kafka shutdown script doesnt seem
> to work for me and I assumed the kill -15 command should trigger a shutdown
> in kafka (I saw messages like kafka shutdown in the logs too)
> On 4 Oct 2013 18:55, "Neha Narkhede" <ne...@gmail.com> wrote:
>
> > When a broker starts up, it receives a LeaderAndIsrRequest from the
> > controller broker telling the broker which partitions it should host and
> > either lead or follow those partitions. If clients send requests to the
> > broker before it has received this request, it throws this error you see.
> > Did you restart the broker very quickly?
> >
> > You can mitigate this issue by using controlled shutdown to stop a
> broker.
> >
> > Thanks,
> > Neha
> > On Oct 4, 2013 2:45 AM, "Aniket Bhatnagar" <an...@gmail.com>
> > wrote:
> >
> > > Because Kafka was detecting localhost.domain as hostname, I commented
> out
> > > the line "127.0.0.1 localhost.localdomain localhost" and
> > added
> > > "127.0.0.1 ip-10-0-1-20.localdomain" in etc/hosts. When I restart Kafka
> > > (issues kill -15 pid), writes to existing topics are failing and I see
> > > several of the following errors:
> > >
> > > [2013-10-04 03:43:23,565] WARN [KafkaApi-1] Produce request with
> > > correlation id 12868451 from client on partition
> > > [consumertracking.gnomes.4002,5] failed due to Partition
> > > [consumertracking.gnomes.4002,5] doesn't exist on 1
> > > (kafka.server.KafkaApis)
> > >
> > >
> > > I have checked the kafka data directory and all files are intact. What
> > > could have caused this issue and how can it be fixed?
> > >
> >
>
Re: Kafka stops accepting messages for existing topics after a
hostname change and restart
Posted by Aniket Bhatnagar <an...@gmail.com>.
I did restart broker very quickly. I saw similar errors for about 5 mins
and that's when I decided to shutdown all kafka brokers and start them one
by one. That seems to have enabled writes in kafka instantly after brokers
were back up.
How do I do a controlled shutdown? The kafka shutdown script doesnt seem
to work for me and I assumed the kill -15 command should trigger a shutdown
in kafka (I saw messages like kafka shutdown in the logs too)
On 4 Oct 2013 18:55, "Neha Narkhede" <ne...@gmail.com> wrote:
> When a broker starts up, it receives a LeaderAndIsrRequest from the
> controller broker telling the broker which partitions it should host and
> either lead or follow those partitions. If clients send requests to the
> broker before it has received this request, it throws this error you see.
> Did you restart the broker very quickly?
>
> You can mitigate this issue by using controlled shutdown to stop a broker.
>
> Thanks,
> Neha
> On Oct 4, 2013 2:45 AM, "Aniket Bhatnagar" <an...@gmail.com>
> wrote:
>
> > Because Kafka was detecting localhost.domain as hostname, I commented out
> > the line "127.0.0.1 localhost.localdomain localhost" and
> added
> > "127.0.0.1 ip-10-0-1-20.localdomain" in etc/hosts. When I restart Kafka
> > (issues kill -15 pid), writes to existing topics are failing and I see
> > several of the following errors:
> >
> > [2013-10-04 03:43:23,565] WARN [KafkaApi-1] Produce request with
> > correlation id 12868451 from client on partition
> > [consumertracking.gnomes.4002,5] failed due to Partition
> > [consumertracking.gnomes.4002,5] doesn't exist on 1
> > (kafka.server.KafkaApis)
> >
> >
> > I have checked the kafka data directory and all files are intact. What
> > could have caused this issue and how can it be fixed?
> >
>
Re: Kafka stops accepting messages for existing topics after a
hostname change and restart
Posted by Neha Narkhede <ne...@gmail.com>.
When a broker starts up, it receives a LeaderAndIsrRequest from the
controller broker telling the broker which partitions it should host and
either lead or follow those partitions. If clients send requests to the
broker before it has received this request, it throws this error you see.
Did you restart the broker very quickly?
You can mitigate this issue by using controlled shutdown to stop a broker.
Thanks,
Neha
On Oct 4, 2013 2:45 AM, "Aniket Bhatnagar" <an...@gmail.com>
wrote:
> Because Kafka was detecting localhost.domain as hostname, I commented out
> the line "127.0.0.1 localhost.localdomain localhost" and added
> "127.0.0.1 ip-10-0-1-20.localdomain" in etc/hosts. When I restart Kafka
> (issues kill -15 pid), writes to existing topics are failing and I see
> several of the following errors:
>
> [2013-10-04 03:43:23,565] WARN [KafkaApi-1] Produce request with
> correlation id 12868451 from client on partition
> [consumertracking.gnomes.4002,5] failed due to Partition
> [consumertracking.gnomes.4002,5] doesn't exist on 1
> (kafka.server.KafkaApis)
>
>
> I have checked the kafka data directory and all files are intact. What
> could have caused this issue and how can it be fixed?
>