You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kane Kane <ka...@gmail.com> on 2013/11/01 09:51:22 UTC
uncontrolled shutdown
When machine with kafka dies, most often broker cannot start itself with
errors about index files being corrupt. After i delete them manually broker
usually can boot up successfully. Shouldn't kafka try to delete/rebuild
broken index files itself?
Also this exception looks a bit weird:
java.lang.IllegalArgumentException: requirement failed: Corrupt index
found, index file
(/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has non-zero
size but the last offset is 385258 and the base offset is 385258
Thanks.
Re: uncontrolled shutdown
Posted by Kane Kane <ka...@gmail.com>.
Hello, yes, I can easily reproduce it, will send it to you asap.
Thanks.
On Fri, Nov 1, 2013 at 2:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Kane, do you mind attach your index file causing the issue if you
> still have it?
>
> Guozhang
>
>
> On Fri, Nov 1, 2013 at 8:42 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Currently the index files will only be deleted on startup if there are
> any
> > .swap file indicating the server crashed while opening the log segments.
> We
> > should probably change this logic.
> >
> > Guozhang
> >
> >
> > On Fri, Nov 1, 2013 at 8:16 AM, Kane Kane <ka...@gmail.com> wrote:
> >
> >> Filed a bug meanwhile, https://issues.apache.org/jira/browse/KAFKA-1112
> >>
> >>
> >> On Fri, Nov 1, 2013 at 8:15 AM, Neha Narkhede <neha.narkhede@gmail.com
> >> >wrote:
> >>
> >> > Jun,
> >> >
> >> > I've seen this happen in the latest code in trunk, need to remember
> how
> >> to
> >> > reproduce it.
> >> >
> >> > Thanks,
> >> > Neha
> >> > On Nov 1, 2013 7:59 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >> >
> >> > > Are you using the latest code in the 0.8 branch?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com>
> >> wrote:
> >> > >
> >> > > > Neha, yes when i kill it with -9, sure I will file a bug.
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > > >
> >> > > > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <
> >> neha.narkhede@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > When you say the Kafka dies, did you mean a kill -9 or kill -15
> ?
> >> I
> >> > > agree
> >> > > > > that Kafka should try and delete/rebuild index files, but in
> this
> >> > case,
> >> > > > it
> >> > > > > sounds like a bug. Can you file a JIRA?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Neha
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <
> kane.isturm@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > When machine with kafka dies, most often broker cannot start
> >> itself
> >> > > > with
> >> > > > > > errors about index files being corrupt. After i delete them
> >> > manually
> >> > > > > broker
> >> > > > > > usually can boot up successfully. Shouldn't kafka try to
> >> > > delete/rebuild
> >> > > > > > broken index files itself?
> >> > > > > >
> >> > > > > > Also this exception looks a bit weird:
> >> > > > > > java.lang.IllegalArgumentException: requirement failed:
> Corrupt
> >> > index
> >> > > > > > found, index file
> >> > > > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
> >> > > non-zero
> >> > > > > > size but the last offset is 385258 and the base offset is
> 385258
> >> > > > > >
> >> > > > > > Thanks.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
Re: uncontrolled shutdown
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Kane, do you mind attach your index file causing the issue if you
still have it?
Guozhang
On Fri, Nov 1, 2013 at 8:42 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Currently the index files will only be deleted on startup if there are any
> .swap file indicating the server crashed while opening the log segments. We
> should probably change this logic.
>
> Guozhang
>
>
> On Fri, Nov 1, 2013 at 8:16 AM, Kane Kane <ka...@gmail.com> wrote:
>
>> Filed a bug meanwhile, https://issues.apache.org/jira/browse/KAFKA-1112
>>
>>
>> On Fri, Nov 1, 2013 at 8:15 AM, Neha Narkhede <neha.narkhede@gmail.com
>> >wrote:
>>
>> > Jun,
>> >
>> > I've seen this happen in the latest code in trunk, need to remember how
>> to
>> > reproduce it.
>> >
>> > Thanks,
>> > Neha
>> > On Nov 1, 2013 7:59 AM, "Jun Rao" <ju...@gmail.com> wrote:
>> >
>> > > Are you using the latest code in the 0.8 branch?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com>
>> wrote:
>> > >
>> > > > Neha, yes when i kill it with -9, sure I will file a bug.
>> > > >
>> > > > Thanks.
>> > > >
>> > > >
>> > > > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > When you say the Kafka dies, did you mean a kill -9 or kill -15 ?
>> I
>> > > agree
>> > > > > that Kafka should try and delete/rebuild index files, but in this
>> > case,
>> > > > it
>> > > > > sounds like a bug. Can you file a JIRA?
>> > > > >
>> > > > > Thanks,
>> > > > > Neha
>> > > > >
>> > > > >
>> > > > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > When machine with kafka dies, most often broker cannot start
>> itself
>> > > > with
>> > > > > > errors about index files being corrupt. After i delete them
>> > manually
>> > > > > broker
>> > > > > > usually can boot up successfully. Shouldn't kafka try to
>> > > delete/rebuild
>> > > > > > broken index files itself?
>> > > > > >
>> > > > > > Also this exception looks a bit weird:
>> > > > > > java.lang.IllegalArgumentException: requirement failed: Corrupt
>> > index
>> > > > > > found, index file
>> > > > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
>> > > non-zero
>> > > > > > size but the last offset is 385258 and the base offset is 385258
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: uncontrolled shutdown
Posted by Guozhang Wang <wa...@gmail.com>.
Currently the index files will only be deleted on startup if there are any
.swap file indicating the server crashed while opening the log segments. We
should probably change this logic.
Guozhang
On Fri, Nov 1, 2013 at 8:16 AM, Kane Kane <ka...@gmail.com> wrote:
> Filed a bug meanwhile, https://issues.apache.org/jira/browse/KAFKA-1112
>
>
> On Fri, Nov 1, 2013 at 8:15 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Jun,
> >
> > I've seen this happen in the latest code in trunk, need to remember how
> to
> > reproduce it.
> >
> > Thanks,
> > Neha
> > On Nov 1, 2013 7:59 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >
> > > Are you using the latest code in the 0.8 branch?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com>
> wrote:
> > >
> > > > Neha, yes when i kill it with -9, sure I will file a bug.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I
> > > agree
> > > > > that Kafka should try and delete/rebuild index files, but in this
> > case,
> > > > it
> > > > > sounds like a bug. Can you file a JIRA?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com>
> > > wrote:
> > > > >
> > > > > > When machine with kafka dies, most often broker cannot start
> itself
> > > > with
> > > > > > errors about index files being corrupt. After i delete them
> > manually
> > > > > broker
> > > > > > usually can boot up successfully. Shouldn't kafka try to
> > > delete/rebuild
> > > > > > broken index files itself?
> > > > > >
> > > > > > Also this exception looks a bit weird:
> > > > > > java.lang.IllegalArgumentException: requirement failed: Corrupt
> > index
> > > > > > found, index file
> > > > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
> > > non-zero
> > > > > > size but the last offset is 385258 and the base offset is 385258
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
-- Guozhang
Re: uncontrolled shutdown
Posted by Kane Kane <ka...@gmail.com>.
Filed a bug meanwhile, https://issues.apache.org/jira/browse/KAFKA-1112
On Fri, Nov 1, 2013 at 8:15 AM, Neha Narkhede <ne...@gmail.com>wrote:
> Jun,
>
> I've seen this happen in the latest code in trunk, need to remember how to
> reproduce it.
>
> Thanks,
> Neha
> On Nov 1, 2013 7:59 AM, "Jun Rao" <ju...@gmail.com> wrote:
>
> > Are you using the latest code in the 0.8 branch?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com> wrote:
> >
> > > Neha, yes when i kill it with -9, sure I will file a bug.
> > >
> > > Thanks.
> > >
> > >
> > > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I
> > agree
> > > > that Kafka should try and delete/rebuild index files, but in this
> case,
> > > it
> > > > sounds like a bug. Can you file a JIRA?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com>
> > wrote:
> > > >
> > > > > When machine with kafka dies, most often broker cannot start itself
> > > with
> > > > > errors about index files being corrupt. After i delete them
> manually
> > > > broker
> > > > > usually can boot up successfully. Shouldn't kafka try to
> > delete/rebuild
> > > > > broken index files itself?
> > > > >
> > > > > Also this exception looks a bit weird:
> > > > > java.lang.IllegalArgumentException: requirement failed: Corrupt
> index
> > > > > found, index file
> > > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
> > non-zero
> > > > > size but the last offset is 385258 and the base offset is 385258
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > >
> >
>
Re: uncontrolled shutdown
Posted by Neha Narkhede <ne...@gmail.com>.
Jun,
I've seen this happen in the latest code in trunk, need to remember how to
reproduce it.
Thanks,
Neha
On Nov 1, 2013 7:59 AM, "Jun Rao" <ju...@gmail.com> wrote:
> Are you using the latest code in the 0.8 branch?
>
> Thanks,
>
> Jun
>
>
> On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com> wrote:
>
> > Neha, yes when i kill it with -9, sure I will file a bug.
> >
> > Thanks.
> >
> >
> > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I
> agree
> > > that Kafka should try and delete/rebuild index files, but in this case,
> > it
> > > sounds like a bug. Can you file a JIRA?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com>
> wrote:
> > >
> > > > When machine with kafka dies, most often broker cannot start itself
> > with
> > > > errors about index files being corrupt. After i delete them manually
> > > broker
> > > > usually can boot up successfully. Shouldn't kafka try to
> delete/rebuild
> > > > broken index files itself?
> > > >
> > > > Also this exception looks a bit weird:
> > > > java.lang.IllegalArgumentException: requirement failed: Corrupt index
> > > > found, index file
> > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
> non-zero
> > > > size but the last offset is 385258 and the base offset is 385258
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: uncontrolled shutdown
Posted by Kane Kane <ka...@gmail.com>.
Hello Jun Rao, not it's not the head, I've compiled it a couple of weeks
ago. Should i try with latest?
On Fri, Nov 1, 2013 at 7:58 AM, Jun Rao <ju...@gmail.com> wrote:
> Are you using the latest code in the 0.8 branch?
>
> Thanks,
>
> Jun
>
>
> On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com> wrote:
>
> > Neha, yes when i kill it with -9, sure I will file a bug.
> >
> > Thanks.
> >
> >
> > On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I
> agree
> > > that Kafka should try and delete/rebuild index files, but in this case,
> > it
> > > sounds like a bug. Can you file a JIRA?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com>
> wrote:
> > >
> > > > When machine with kafka dies, most often broker cannot start itself
> > with
> > > > errors about index files being corrupt. After i delete them manually
> > > broker
> > > > usually can boot up successfully. Shouldn't kafka try to
> delete/rebuild
> > > > broken index files itself?
> > > >
> > > > Also this exception looks a bit weird:
> > > > java.lang.IllegalArgumentException: requirement failed: Corrupt index
> > > > found, index file
> > > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has
> non-zero
> > > > size but the last offset is 385258 and the base offset is 385258
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: uncontrolled shutdown
Posted by Jun Rao <ju...@gmail.com>.
Are you using the latest code in the 0.8 branch?
Thanks,
Jun
On Fri, Nov 1, 2013 at 7:36 AM, Kane Kane <ka...@gmail.com> wrote:
> Neha, yes when i kill it with -9, sure I will file a bug.
>
> Thanks.
>
>
> On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I agree
> > that Kafka should try and delete/rebuild index files, but in this case,
> it
> > sounds like a bug. Can you file a JIRA?
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com> wrote:
> >
> > > When machine with kafka dies, most often broker cannot start itself
> with
> > > errors about index files being corrupt. After i delete them manually
> > broker
> > > usually can boot up successfully. Shouldn't kafka try to delete/rebuild
> > > broken index files itself?
> > >
> > > Also this exception looks a bit weird:
> > > java.lang.IllegalArgumentException: requirement failed: Corrupt index
> > > found, index file
> > > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has non-zero
> > > size but the last offset is 385258 and the base offset is 385258
> > >
> > > Thanks.
> > >
> >
>
Re: uncontrolled shutdown
Posted by Kane Kane <ka...@gmail.com>.
Neha, yes when i kill it with -9, sure I will file a bug.
Thanks.
On Fri, Nov 1, 2013 at 3:43 AM, Neha Narkhede <ne...@gmail.com>wrote:
> When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I agree
> that Kafka should try and delete/rebuild index files, but in this case, it
> sounds like a bug. Can you file a JIRA?
>
> Thanks,
> Neha
>
>
> On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com> wrote:
>
> > When machine with kafka dies, most often broker cannot start itself with
> > errors about index files being corrupt. After i delete them manually
> broker
> > usually can boot up successfully. Shouldn't kafka try to delete/rebuild
> > broken index files itself?
> >
> > Also this exception looks a bit weird:
> > java.lang.IllegalArgumentException: requirement failed: Corrupt index
> > found, index file
> > (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has non-zero
> > size but the last offset is 385258 and the base offset is 385258
> >
> > Thanks.
> >
>
Re: uncontrolled shutdown
Posted by Neha Narkhede <ne...@gmail.com>.
When you say the Kafka dies, did you mean a kill -9 or kill -15 ? I agree
that Kafka should try and delete/rebuild index files, but in this case, it
sounds like a bug. Can you file a JIRA?
Thanks,
Neha
On Fri, Nov 1, 2013 at 1:51 AM, Kane Kane <ka...@gmail.com> wrote:
> When machine with kafka dies, most often broker cannot start itself with
> errors about index files being corrupt. After i delete them manually broker
> usually can boot up successfully. Shouldn't kafka try to delete/rebuild
> broken index files itself?
>
> Also this exception looks a bit weird:
> java.lang.IllegalArgumentException: requirement failed: Corrupt index
> found, index file
> (/disk1/kafka-logs/benchmark-2/00000000000000385258.index) has non-zero
> size but the last offset is 385258 and the base offset is 385258
>
> Thanks.
>