RE: Topics getting corrupted

["Bateman, Matt" <ma...@ebay.com>]

Hi Guys,

We're getting this same issue. It seems that this thread petered out with the suggestion to try 0.7.

Is there anything we can try with 0.6 to fix this? 

Thanks,

Matt

-----Original Message-----
From: Nalin DeZoysa [mailto:ndezoysa@tagged.com] 
Sent: Thursday, November 03, 2011 12:53 PM
To: kafka-users@incubator.apache.org
Subject: Re: Topics getting corrupted

Hi Neha,

No this isn't a valid offset. Right now its not possible to test with 0.7 because of the new message format, and i'm using a node.js kafka client which doesn't support this yet. Do you know of a workaround that i can use until I can test with the new version to get it to work/recover ? I'm running with Kafka 0.6



Thanks,
Nalin

On Thu, Nov 3, 2011 at 12:28 PM, Neha Narkhede <ne...@gmail.com>wrote:

> Nalin,
>
> >> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is 
> >> out
> of range
>
> Is this offset a valid one ? If you check the logs for that topic, do 
> you see this offset ?
>
> If yes, then it could be a bug in the message set iterator.
>
> >> > also the command that worked for me was ./bin/kafka-run-class.sh
> kafka.utils.DumpLogSegments log_file
>
> That means, you are running on an older version. Do you mind trying 
> this out with 0.7 RC5 ?
>
> Thanks,
> Neha
>
> On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> > Ran the util, log looks fine. Seems like the issue isn't with the 
> > log
> file
> > itself. The issue does not resolve itself even though i restart 
> > kafka and zookeeper, or run kafka without zookeeper.
> >
> > also the command that worked for me was ./bin/kafka-run-class.sh 
> > kafka.utils.DumpLogSegments log_file
> >
> > -Nalin
> >
> > On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede 
> ><neha.narkhede@gmail.com
> >wrote:
> >
> >> Nalin,
> >>
> >> You can use it like this -
> >> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file 
> >> --noprint
> >>
> >> This will print the offset of each message, along with other attributes.
> >>
> >> We haven't really standardized this tool, in spite of being very useful.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa 
> >> <nd...@tagged.com>
> >> wrote:
> >> > Could you tell me how to run DumpLogSegment?
> >> >
> >> >
> >> > Thanks,
> >> > Nalin
> >> >
> >> > On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
> >> >
> >> >> I don't see the attached log file. I guess the Apache mailing 
> >> >> list
> >> doesn't
> >> >> really allow attachments. Could you run the util 
> >> >> DumpLogSegement? It
> >> will
> >> >> tell you whether the log file is corrupted or not.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa 
> >> >> <nd...@tagged.com>
> >> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I'm seeing that certain topics get corrupted - I can publish 
> >> >> > to it,
> >> but
> >> >> > when I try and subscribe to a topic, the response that comes 
> >> >> > back
> has
> >> >> > header error code 1. I also see this exception being thrown by
> Kafka.
> >> >> >
> >> >> > [2011-11-02 15:36:40,947] ERROR error when processing request 
> >> >> > topic:cards_card_251791, part:0 offset:281474976724790
> maxSize:1048576
> >> >> > (kafka.server.KafkaRequestHandlers)
> >> >> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 
> >> >> > is
> out
> >> of
> >> >> > range
> >> >> >  at kafka.log.Log$.findRange(Log.scala:47)
> >> >> > at kafka.log.Log.read(Log.scala:223)  at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$r
> eadMessageSet(KafkaRequestHandlers.scala:124)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandl
> ers.scala:107)
> >> >> >  at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
> uestHandlers.scala:42)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
> uestHandlers.scala:42)
> >> >> >  at kafka.network.Processor.handle(SocketServer.scala:268)
> >> >> > at kafka.network.Processor.read(SocketServer.scala:291)
> >> >> >  at kafka.network.Processor.run(SocketServer.scala:202)
> >> >> > at java.lang.Thread.run(Thread.java:619)
> >> >> >
> >> >> >
> >> >> >
> >> >> > I have attached the log file for the topic.
> >> >> >
> >> >> > Is there a way to work around this issue?
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > Nalin
> >> >> >
> >> >>
> >> >
> >>
> >
>

Re: Topics getting corrupted

[Taylor Gautier <tg...@tagged.com>]

I'm pretty sure all the issues we ran into we're caused by us. We have
been developing a NodeJS client and this was the source of most
issues.

As of about two weeks ago we made some final fixes to this client and
the tier has been rock solid. We have moved ourselves to Kafka 0.7.



On Jan 24, 2012, at 6:45 PM, "Bateman, Matt" <ma...@ebay.com> wrote:

> Hi Guys,
>
> We're getting this same issue. It seems that this thread petered out with the suggestion to try 0.7.
>
> Is there anything we can try with 0.6 to fix this?
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Nalin DeZoysa [mailto:ndezoysa@tagged.com]
> Sent: Thursday, November 03, 2011 12:53 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Topics getting corrupted
>
> Hi Neha,
>
> No this isn't a valid offset. Right now its not possible to test with 0.7 because of the new message format, and i'm using a node.js kafka client which doesn't support this yet. Do you know of a workaround that i can use until I can test with the new version to get it to work/recover ? I'm running with Kafka 0.6
>
>
>
> Thanks,
> Nalin
>
> On Thu, Nov 3, 2011 at 12:28 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Nalin,
>>
>>>> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is
>>>> out
>> of range
>>
>> Is this offset a valid one ? If you check the logs for that topic, do
>> you see this offset ?
>>
>> If yes, then it could be a bug in the message set iterator.
>>
>>>>> also the command that worked for me was ./bin/kafka-run-class.sh
>> kafka.utils.DumpLogSegments log_file
>>
>> That means, you are running on an older version. Do you mind trying
>> this out with 0.7 RC5 ?
>>
>> Thanks,
>> Neha
>>
>> On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com>
>> wrote:
>>> Ran the util, log looks fine. Seems like the issue isn't with the
>>> log
>> file
>>> itself. The issue does not resolve itself even though i restart
>>> kafka and zookeeper, or run kafka without zookeeper.
>>>
>>> also the command that worked for me was ./bin/kafka-run-class.sh
>>> kafka.utils.DumpLogSegments log_file
>>>
>>> -Nalin
>>>
>>> On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede
>>> <neha.narkhede@gmail.com
>>> wrote:
>>>
>>>> Nalin,
>>>>
>>>> You can use it like this -
>>>> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file
>>>> --noprint
>>>>
>>>> This will print the offset of each message, along with other attributes.
>>>>
>>>> We haven't really standardized this tool, in spite of being very useful.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa
>>>> <nd...@tagged.com>
>>>> wrote:
>>>>> Could you tell me how to run DumpLogSegment?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Nalin
>>>>>
>>>>> On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>>
>>>>>> I don't see the attached log file. I guess the Apache mailing
>>>>>> list
>>>> doesn't
>>>>>> really allow attachments. Could you run the util
>>>>>> DumpLogSegement? It
>>>> will
>>>>>> tell you whether the log file is corrupted or not.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa
>>>>>> <nd...@tagged.com>
>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm seeing that certain topics get corrupted - I can publish
>>>>>>> to it,
>>>> but
>>>>>>> when I try and subscribe to a topic, the response that comes
>>>>>>> back
>> has
>>>>>>> header error code 1. I also see this exception being thrown by
>> Kafka.
>>>>>>>
>>>>>>> [2011-11-02 15:36:40,947] ERROR error when processing request
>>>>>>> topic:cards_card_251791, part:0 offset:281474976724790
>> maxSize:1048576
>>>>>>> (kafka.server.KafkaRequestHandlers)
>>>>>>> kafka.common.OffsetOutOfRangeException: offset 281474976724790
>>>>>>> is
>> out
>>>> of
>>>>>>> range
>>>>>>> at kafka.log.Log$.findRange(Log.scala:47)
>>>>>>> at kafka.log.Log.read(Log.scala:223)  at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$r
>> eadMessageSet(KafkaRequestHandlers.scala:124)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandl
>> ers.scala:107)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
>> uestHandlers.scala:42)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
>> uestHandlers.scala:42)
>>>>>>> at kafka.network.Processor.handle(SocketServer.scala:268)
>>>>>>> at kafka.network.Processor.read(SocketServer.scala:291)
>>>>>>> at kafka.network.Processor.run(SocketServer.scala:202)
>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have attached the log file for the topic.
>>>>>>>
>>>>>>> Is there a way to work around this issue?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nalin
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>