You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nalin DeZoysa <nd...@tagged.com> on 2011/11/02 23:54:32 UTC
Topics getting corrupted
Hi,
I'm seeing that certain topics get corrupted - I can publish to it, but
when I try and subscribe to a topic, the response that comes back has
header error code 1. I also see this exception being thrown by Kafka.
[2011-11-02 15:36:40,947] ERROR error when processing request
topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
(kafka.server.KafkaRequestHandlers)
kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out of
range
at kafka.log.Log$.findRange(Log.scala:47)
at kafka.log.Log.read(Log.scala:223)
at
kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
at
kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
at kafka.network.Processor.handle(SocketServer.scala:268)
at kafka.network.Processor.read(SocketServer.scala:291)
at kafka.network.Processor.run(SocketServer.scala:202)
at java.lang.Thread.run(Thread.java:619)
I have attached the log file for the topic.
Is there a way to work around this issue?
Thanks,
Nalin
Re: Topics getting corrupted
Posted by Taylor Gautier <tg...@tagged.com>.
Done - https://issues.apache.org/jira/browse/KAFKA-189
On Fri, Nov 4, 2011 at 3:34 PM, Jay Kreps <ja...@gmail.com> wrote:
> That is a great point, would you be willing to file a JIRA for that?
>
> -Jay
>
> On Thu, Nov 3, 2011 at 10:06 PM, Taylor Gautier <tg...@tagged.com>
> wrote:
>
> > We just found the issue!
> >
> > It was an error in the Node library we've been working with/updating -
> > seems to be formatting the binary packet wrong when encoding the offset.
> > The dump segment tool was really useful to help confirm that the problem
> > was external - thanks for the pointer to that and the responsiveness on
> the
> > list and IRC.
> >
> > I probably could have used dump segment several times in the past - it'd
> be
> > great to have a script wrapped around it.
> >
> > On Thu, Nov 3, 2011 at 9:58 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > If you are not using zk, are you using SimpleConsumer then?
> > >
> > > Jun
> > >
> > > On Thu, Nov 3, 2011 at 4:31 PM, Taylor Gautier <tg...@tagged.com>
> > > wrote:
> > >
> > > > I'm working with Nalin on this…the offsets in the segment are really
> > low.
> > > > The client is submitting an offset that appears to be within the
> > proper
> > > > range and then when kafka gets it for some reason it gives an error
> > with
> > > > this huge number. We will double check that the offset provided is
> > > exactly
> > > > correct.
> > > >
> > > > Also there was not any zk involved, we've turned it off for the time
> > > being.
> > > >
> > > > On Thu, Nov 3, 2011 at 4:24 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Nalin,
> > > > >
> > > > > It sounds like the consumer computed a wrong offset. At this
> moment,
> > > you
> > > > > will have to either manually change the offset in ZK to a valid one
> > > (use
> > > > > any offset returned in the DumpLogSegement tool), or use a new
> > consumer
> > > > > group (which will consume from either the head or the tail of the
> > > queue).
> > > > >
> > > > > Jun
> > > > >
> > > >
> > >
> >
>
Re: Topics getting corrupted
Posted by Jay Kreps <ja...@gmail.com>.
That is a great point, would you be willing to file a JIRA for that?
-Jay
On Thu, Nov 3, 2011 at 10:06 PM, Taylor Gautier <tg...@tagged.com> wrote:
> We just found the issue!
>
> It was an error in the Node library we've been working with/updating -
> seems to be formatting the binary packet wrong when encoding the offset.
> The dump segment tool was really useful to help confirm that the problem
> was external - thanks for the pointer to that and the responsiveness on the
> list and IRC.
>
> I probably could have used dump segment several times in the past - it'd be
> great to have a script wrapped around it.
>
> On Thu, Nov 3, 2011 at 9:58 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > If you are not using zk, are you using SimpleConsumer then?
> >
> > Jun
> >
> > On Thu, Nov 3, 2011 at 4:31 PM, Taylor Gautier <tg...@tagged.com>
> > wrote:
> >
> > > I'm working with Nalin on this…the offsets in the segment are really
> low.
> > > The client is submitting an offset that appears to be within the
> proper
> > > range and then when kafka gets it for some reason it gives an error
> with
> > > this huge number. We will double check that the offset provided is
> > exactly
> > > correct.
> > >
> > > Also there was not any zk involved, we've turned it off for the time
> > being.
> > >
> > > On Thu, Nov 3, 2011 at 4:24 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Nalin,
> > > >
> > > > It sounds like the consumer computed a wrong offset. At this moment,
> > you
> > > > will have to either manually change the offset in ZK to a valid one
> > (use
> > > > any offset returned in the DumpLogSegement tool), or use a new
> consumer
> > > > group (which will consume from either the head or the tail of the
> > queue).
> > > >
> > > > Jun
> > > >
> > >
> >
>
Re: Topics getting corrupted
Posted by Taylor Gautier <tg...@tagged.com>.
We just found the issue!
It was an error in the Node library we've been working with/updating -
seems to be formatting the binary packet wrong when encoding the offset.
The dump segment tool was really useful to help confirm that the problem
was external - thanks for the pointer to that and the responsiveness on the
list and IRC.
I probably could have used dump segment several times in the past - it'd be
great to have a script wrapped around it.
On Thu, Nov 3, 2011 at 9:58 PM, Jun Rao <ju...@gmail.com> wrote:
> If you are not using zk, are you using SimpleConsumer then?
>
> Jun
>
> On Thu, Nov 3, 2011 at 4:31 PM, Taylor Gautier <tg...@tagged.com>
> wrote:
>
> > I'm working with Nalin on this…the offsets in the segment are really low.
> > The client is submitting an offset that appears to be within the proper
> > range and then when kafka gets it for some reason it gives an error with
> > this huge number. We will double check that the offset provided is
> exactly
> > correct.
> >
> > Also there was not any zk involved, we've turned it off for the time
> being.
> >
> > On Thu, Nov 3, 2011 at 4:24 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Nalin,
> > >
> > > It sounds like the consumer computed a wrong offset. At this moment,
> you
> > > will have to either manually change the offset in ZK to a valid one
> (use
> > > any offset returned in the DumpLogSegement tool), or use a new consumer
> > > group (which will consume from either the head or the tail of the
> queue).
> > >
> > > Jun
> > >
> >
>
Re: Topics getting corrupted
Posted by Jun Rao <ju...@gmail.com>.
If you are not using zk, are you using SimpleConsumer then?
Jun
On Thu, Nov 3, 2011 at 4:31 PM, Taylor Gautier <tg...@tagged.com> wrote:
> I'm working with Nalin on this…the offsets in the segment are really low.
> The client is submitting an offset that appears to be within the proper
> range and then when kafka gets it for some reason it gives an error with
> this huge number. We will double check that the offset provided is exactly
> correct.
>
> Also there was not any zk involved, we've turned it off for the time being.
>
> On Thu, Nov 3, 2011 at 4:24 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Nalin,
> >
> > It sounds like the consumer computed a wrong offset. At this moment, you
> > will have to either manually change the offset in ZK to a valid one (use
> > any offset returned in the DumpLogSegement tool), or use a new consumer
> > group (which will consume from either the head or the tail of the queue).
> >
> > Jun
> >
>
Re: Topics getting corrupted
Posted by Taylor Gautier <tg...@tagged.com>.
I'm working with Nalin on this…the offsets in the segment are really low.
The client is submitting an offset that appears to be within the proper
range and then when kafka gets it for some reason it gives an error with
this huge number. We will double check that the offset provided is exactly
correct.
Also there was not any zk involved, we've turned it off for the time being.
On Thu, Nov 3, 2011 at 4:24 PM, Jun Rao <ju...@gmail.com> wrote:
> Nalin,
>
> It sounds like the consumer computed a wrong offset. At this moment, you
> will have to either manually change the offset in ZK to a valid one (use
> any offset returned in the DumpLogSegement tool), or use a new consumer
> group (which will consume from either the head or the tail of the queue).
>
> Jun
>
Re: Topics getting corrupted
Posted by Jun Rao <ju...@gmail.com>.
Nalin,
It sounds like the consumer computed a wrong offset. At this moment, you
will have to either manually change the offset in ZK to a valid one (use
any offset returned in the DumpLogSegement tool), or use a new consumer
group (which will consume from either the head or the tail of the queue).
Jun
Re: Topics getting corrupted
Posted by Taylor Gautier <tg...@tagged.com>.
I'm pretty sure all the issues we ran into we're caused by us. We have
been developing a NodeJS client and this was the source of most
issues.
As of about two weeks ago we made some final fixes to this client and
the tier has been rock solid. We have moved ourselves to Kafka 0.7.
On Jan 24, 2012, at 6:45 PM, "Bateman, Matt" <ma...@ebay.com> wrote:
> Hi Guys,
>
> We're getting this same issue. It seems that this thread petered out with the suggestion to try 0.7.
>
> Is there anything we can try with 0.6 to fix this?
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Nalin DeZoysa [mailto:ndezoysa@tagged.com]
> Sent: Thursday, November 03, 2011 12:53 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Topics getting corrupted
>
> Hi Neha,
>
> No this isn't a valid offset. Right now its not possible to test with 0.7 because of the new message format, and i'm using a node.js kafka client which doesn't support this yet. Do you know of a workaround that i can use until I can test with the new version to get it to work/recover ? I'm running with Kafka 0.6
>
>
>
> Thanks,
> Nalin
>
> On Thu, Nov 3, 2011 at 12:28 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Nalin,
>>
>>>> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is
>>>> out
>> of range
>>
>> Is this offset a valid one ? If you check the logs for that topic, do
>> you see this offset ?
>>
>> If yes, then it could be a bug in the message set iterator.
>>
>>>>> also the command that worked for me was ./bin/kafka-run-class.sh
>> kafka.utils.DumpLogSegments log_file
>>
>> That means, you are running on an older version. Do you mind trying
>> this out with 0.7 RC5 ?
>>
>> Thanks,
>> Neha
>>
>> On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com>
>> wrote:
>>> Ran the util, log looks fine. Seems like the issue isn't with the
>>> log
>> file
>>> itself. The issue does not resolve itself even though i restart
>>> kafka and zookeeper, or run kafka without zookeeper.
>>>
>>> also the command that worked for me was ./bin/kafka-run-class.sh
>>> kafka.utils.DumpLogSegments log_file
>>>
>>> -Nalin
>>>
>>> On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede
>>> <neha.narkhede@gmail.com
>>> wrote:
>>>
>>>> Nalin,
>>>>
>>>> You can use it like this -
>>>> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file
>>>> --noprint
>>>>
>>>> This will print the offset of each message, along with other attributes.
>>>>
>>>> We haven't really standardized this tool, in spite of being very useful.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa
>>>> <nd...@tagged.com>
>>>> wrote:
>>>>> Could you tell me how to run DumpLogSegment?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Nalin
>>>>>
>>>>> On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>>
>>>>>> I don't see the attached log file. I guess the Apache mailing
>>>>>> list
>>>> doesn't
>>>>>> really allow attachments. Could you run the util
>>>>>> DumpLogSegement? It
>>>> will
>>>>>> tell you whether the log file is corrupted or not.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa
>>>>>> <nd...@tagged.com>
>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm seeing that certain topics get corrupted - I can publish
>>>>>>> to it,
>>>> but
>>>>>>> when I try and subscribe to a topic, the response that comes
>>>>>>> back
>> has
>>>>>>> header error code 1. I also see this exception being thrown by
>> Kafka.
>>>>>>>
>>>>>>> [2011-11-02 15:36:40,947] ERROR error when processing request
>>>>>>> topic:cards_card_251791, part:0 offset:281474976724790
>> maxSize:1048576
>>>>>>> (kafka.server.KafkaRequestHandlers)
>>>>>>> kafka.common.OffsetOutOfRangeException: offset 281474976724790
>>>>>>> is
>> out
>>>> of
>>>>>>> range
>>>>>>> at kafka.log.Log$.findRange(Log.scala:47)
>>>>>>> at kafka.log.Log.read(Log.scala:223) at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$r
>> eadMessageSet(KafkaRequestHandlers.scala:124)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandl
>> ers.scala:107)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
>> uestHandlers.scala:42)
>>>>>>> at
>>>>>>>
>>>>>>
>>>>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
>> uestHandlers.scala:42)
>>>>>>> at kafka.network.Processor.handle(SocketServer.scala:268)
>>>>>>> at kafka.network.Processor.read(SocketServer.scala:291)
>>>>>>> at kafka.network.Processor.run(SocketServer.scala:202)
>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have attached the log file for the topic.
>>>>>>>
>>>>>>> Is there a way to work around this issue?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nalin
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
RE: Topics getting corrupted
Posted by "Bateman, Matt" <ma...@ebay.com>.
Hi Guys,
We're getting this same issue. It seems that this thread petered out with the suggestion to try 0.7.
Is there anything we can try with 0.6 to fix this?
Thanks,
Matt
-----Original Message-----
From: Nalin DeZoysa [mailto:ndezoysa@tagged.com]
Sent: Thursday, November 03, 2011 12:53 PM
To: kafka-users@incubator.apache.org
Subject: Re: Topics getting corrupted
Hi Neha,
No this isn't a valid offset. Right now its not possible to test with 0.7 because of the new message format, and i'm using a node.js kafka client which doesn't support this yet. Do you know of a workaround that i can use until I can test with the new version to get it to work/recover ? I'm running with Kafka 0.6
Thanks,
Nalin
On Thu, Nov 3, 2011 at 12:28 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Nalin,
>
> >> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is
> >> out
> of range
>
> Is this offset a valid one ? If you check the logs for that topic, do
> you see this offset ?
>
> If yes, then it could be a bug in the message set iterator.
>
> >> > also the command that worked for me was ./bin/kafka-run-class.sh
> kafka.utils.DumpLogSegments log_file
>
> That means, you are running on an older version. Do you mind trying
> this out with 0.7 RC5 ?
>
> Thanks,
> Neha
>
> On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> > Ran the util, log looks fine. Seems like the issue isn't with the
> > log
> file
> > itself. The issue does not resolve itself even though i restart
> > kafka and zookeeper, or run kafka without zookeeper.
> >
> > also the command that worked for me was ./bin/kafka-run-class.sh
> > kafka.utils.DumpLogSegments log_file
> >
> > -Nalin
> >
> > On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede
> ><neha.narkhede@gmail.com
> >wrote:
> >
> >> Nalin,
> >>
> >> You can use it like this -
> >> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file
> >> --noprint
> >>
> >> This will print the offset of each message, along with other attributes.
> >>
> >> We haven't really standardized this tool, in spite of being very useful.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa
> >> <nd...@tagged.com>
> >> wrote:
> >> > Could you tell me how to run DumpLogSegment?
> >> >
> >> >
> >> > Thanks,
> >> > Nalin
> >> >
> >> > On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
> >> >
> >> >> I don't see the attached log file. I guess the Apache mailing
> >> >> list
> >> doesn't
> >> >> really allow attachments. Could you run the util
> >> >> DumpLogSegement? It
> >> will
> >> >> tell you whether the log file is corrupted or not.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa
> >> >> <nd...@tagged.com>
> >> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I'm seeing that certain topics get corrupted - I can publish
> >> >> > to it,
> >> but
> >> >> > when I try and subscribe to a topic, the response that comes
> >> >> > back
> has
> >> >> > header error code 1. I also see this exception being thrown by
> Kafka.
> >> >> >
> >> >> > [2011-11-02 15:36:40,947] ERROR error when processing request
> >> >> > topic:cards_card_251791, part:0 offset:281474976724790
> maxSize:1048576
> >> >> > (kafka.server.KafkaRequestHandlers)
> >> >> > kafka.common.OffsetOutOfRangeException: offset 281474976724790
> >> >> > is
> out
> >> of
> >> >> > range
> >> >> > at kafka.log.Log$.findRange(Log.scala:47)
> >> >> > at kafka.log.Log.read(Log.scala:223) at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$r
> eadMessageSet(KafkaRequestHandlers.scala:124)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandl
> ers.scala:107)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
> uestHandlers.scala:42)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaReq
> uestHandlers.scala:42)
> >> >> > at kafka.network.Processor.handle(SocketServer.scala:268)
> >> >> > at kafka.network.Processor.read(SocketServer.scala:291)
> >> >> > at kafka.network.Processor.run(SocketServer.scala:202)
> >> >> > at java.lang.Thread.run(Thread.java:619)
> >> >> >
> >> >> >
> >> >> >
> >> >> > I have attached the log file for the topic.
> >> >> >
> >> >> > Is there a way to work around this issue?
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > Nalin
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Topics getting corrupted
Posted by Nalin DeZoysa <nd...@tagged.com>.
Hi Neha,
No this isn't a valid offset. Right now its not possible to test with 0.7
because of the new message format, and i'm using a node.js kafka client
which doesn't support this yet. Do you know of a workaround that i can use
until I can test with the new version to get it to work/recover ? I'm
running with Kafka 0.6
Thanks,
Nalin
On Thu, Nov 3, 2011 at 12:28 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Nalin,
>
> >> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out
> of range
>
> Is this offset a valid one ? If you check the logs for that topic, do
> you see this offset ?
>
> If yes, then it could be a bug in the message set iterator.
>
> >> > also the command that worked for me was ./bin/kafka-run-class.sh
> kafka.utils.DumpLogSegments log_file
>
> That means, you are running on an older version. Do you mind trying
> this out with 0.7 RC5 ?
>
> Thanks,
> Neha
>
> On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> > Ran the util, log looks fine. Seems like the issue isn't with the log
> file
> > itself. The issue does not resolve itself even though i restart kafka and
> > zookeeper, or run kafka without zookeeper.
> >
> > also the command that worked for me was ./bin/kafka-run-class.sh
> > kafka.utils.DumpLogSegments log_file
> >
> > -Nalin
> >
> > On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Nalin,
> >>
> >> You can use it like this -
> >> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file --noprint
> >>
> >> This will print the offset of each message, along with other attributes.
> >>
> >> We haven't really standardized this tool, in spite of being very useful.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa <nd...@tagged.com>
> >> wrote:
> >> > Could you tell me how to run DumpLogSegment?
> >> >
> >> >
> >> > Thanks,
> >> > Nalin
> >> >
> >> > On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
> >> >
> >> >> I don't see the attached log file. I guess the Apache mailing list
> >> doesn't
> >> >> really allow attachments. Could you run the util DumpLogSegement? It
> >> will
> >> >> tell you whether the log file is corrupted or not.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com>
> >> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I'm seeing that certain topics get corrupted - I can publish to it,
> >> but
> >> >> > when I try and subscribe to a topic, the response that comes back
> has
> >> >> > header error code 1. I also see this exception being thrown by
> Kafka.
> >> >> >
> >> >> > [2011-11-02 15:36:40,947] ERROR error when processing request
> >> >> > topic:cards_card_251791, part:0 offset:281474976724790
> maxSize:1048576
> >> >> > (kafka.server.KafkaRequestHandlers)
> >> >> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is
> out
> >> of
> >> >> > range
> >> >> > at kafka.log.Log$.findRange(Log.scala:47)
> >> >> > at kafka.log.Log.read(Log.scala:223)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> >> >> > at
> >> >> >
> >> >>
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> >> >> > at kafka.network.Processor.handle(SocketServer.scala:268)
> >> >> > at kafka.network.Processor.read(SocketServer.scala:291)
> >> >> > at kafka.network.Processor.run(SocketServer.scala:202)
> >> >> > at java.lang.Thread.run(Thread.java:619)
> >> >> >
> >> >> >
> >> >> >
> >> >> > I have attached the log file for the topic.
> >> >> >
> >> >> > Is there a way to work around this issue?
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > Nalin
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Topics getting corrupted
Posted by Neha Narkhede <ne...@gmail.com>.
Nalin,
>> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out of range
Is this offset a valid one ? If you check the logs for that topic, do
you see this offset ?
If yes, then it could be a bug in the message set iterator.
>> > also the command that worked for me was ./bin/kafka-run-class.sh kafka.utils.DumpLogSegments log_file
That means, you are running on an older version. Do you mind trying
this out with 0.7 RC5 ?
Thanks,
Neha
On Thu, Nov 3, 2011 at 11:58 AM, Nalin DeZoysa <nd...@tagged.com> wrote:
> Ran the util, log looks fine. Seems like the issue isn't with the log file
> itself. The issue does not resolve itself even though i restart kafka and
> zookeeper, or run kafka without zookeeper.
>
> also the command that worked for me was ./bin/kafka-run-class.sh
> kafka.utils.DumpLogSegments log_file
>
> -Nalin
>
> On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Nalin,
>>
>> You can use it like this -
>> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file --noprint
>>
>> This will print the offset of each message, along with other attributes.
>>
>> We haven't really standardized this tool, in spite of being very useful.
>>
>> Thanks,
>> Neha
>>
>> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa <nd...@tagged.com>
>> wrote:
>> > Could you tell me how to run DumpLogSegment?
>> >
>> >
>> > Thanks,
>> > Nalin
>> >
>> > On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
>> >
>> >> I don't see the attached log file. I guess the Apache mailing list
>> doesn't
>> >> really allow attachments. Could you run the util DumpLogSegement? It
>> will
>> >> tell you whether the log file is corrupted or not.
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com>
>> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > I'm seeing that certain topics get corrupted - I can publish to it,
>> but
>> >> > when I try and subscribe to a topic, the response that comes back has
>> >> > header error code 1. I also see this exception being thrown by Kafka.
>> >> >
>> >> > [2011-11-02 15:36:40,947] ERROR error when processing request
>> >> > topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
>> >> > (kafka.server.KafkaRequestHandlers)
>> >> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out
>> of
>> >> > range
>> >> > at kafka.log.Log$.findRange(Log.scala:47)
>> >> > at kafka.log.Log.read(Log.scala:223)
>> >> > at
>> >> >
>> >>
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
>> >> > at
>> >> >
>> >>
>> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
>> >> > at
>> >> >
>> >>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
>> >> > at
>> >> >
>> >>
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
>> >> > at kafka.network.Processor.handle(SocketServer.scala:268)
>> >> > at kafka.network.Processor.read(SocketServer.scala:291)
>> >> > at kafka.network.Processor.run(SocketServer.scala:202)
>> >> > at java.lang.Thread.run(Thread.java:619)
>> >> >
>> >> >
>> >> >
>> >> > I have attached the log file for the topic.
>> >> >
>> >> > Is there a way to work around this issue?
>> >> >
>> >> >
>> >> > Thanks,
>> >> > Nalin
>> >> >
>> >>
>> >
>>
>
Re: Topics getting corrupted
Posted by Nalin DeZoysa <nd...@tagged.com>.
Ran the util, log looks fine. Seems like the issue isn't with the log file
itself. The issue does not resolve itself even though i restart kafka and
zookeeper, or run kafka without zookeeper.
also the command that worked for me was ./bin/kafka-run-class.sh
kafka.utils.DumpLogSegments log_file
-Nalin
On Thu, Nov 3, 2011 at 11:40 AM, Neha Narkhede <ne...@gmail.com>wrote:
> Nalin,
>
> You can use it like this -
> ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file --noprint
>
> This will print the offset of each message, along with other attributes.
>
> We haven't really standardized this tool, in spite of being very useful.
>
> Thanks,
> Neha
>
> On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> > Could you tell me how to run DumpLogSegment?
> >
> >
> > Thanks,
> > Nalin
> >
> > On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> I don't see the attached log file. I guess the Apache mailing list
> doesn't
> >> really allow attachments. Could you run the util DumpLogSegement? It
> will
> >> tell you whether the log file is corrupted or not.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm seeing that certain topics get corrupted - I can publish to it,
> but
> >> > when I try and subscribe to a topic, the response that comes back has
> >> > header error code 1. I also see this exception being thrown by Kafka.
> >> >
> >> > [2011-11-02 15:36:40,947] ERROR error when processing request
> >> > topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
> >> > (kafka.server.KafkaRequestHandlers)
> >> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out
> of
> >> > range
> >> > at kafka.log.Log$.findRange(Log.scala:47)
> >> > at kafka.log.Log.read(Log.scala:223)
> >> > at
> >> >
> >>
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> >> > at
> >> >
> >>
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
> >> > at
> >> >
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> >> > at
> >> >
> >>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> >> > at kafka.network.Processor.handle(SocketServer.scala:268)
> >> > at kafka.network.Processor.read(SocketServer.scala:291)
> >> > at kafka.network.Processor.run(SocketServer.scala:202)
> >> > at java.lang.Thread.run(Thread.java:619)
> >> >
> >> >
> >> >
> >> > I have attached the log file for the topic.
> >> >
> >> > Is there a way to work around this issue?
> >> >
> >> >
> >> > Thanks,
> >> > Nalin
> >> >
> >>
> >
>
Re: Topics getting corrupted
Posted by Neha Narkhede <ne...@gmail.com>.
Nalin,
You can use it like this -
./bin/kafka-run-class.sh kafka.tools.DumpLogSegments log_file --noprint
This will print the offset of each message, along with other attributes.
We haven't really standardized this tool, in spite of being very useful.
Thanks,
Neha
On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa <nd...@tagged.com> wrote:
> Could you tell me how to run DumpLogSegment?
>
>
> Thanks,
> Nalin
>
> On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
>
>> I don't see the attached log file. I guess the Apache mailing list doesn't
>> really allow attachments. Could you run the util DumpLogSegement? It will
>> tell you whether the log file is corrupted or not.
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com> wrote:
>>
>> > Hi,
>> >
>> > I'm seeing that certain topics get corrupted - I can publish to it, but
>> > when I try and subscribe to a topic, the response that comes back has
>> > header error code 1. I also see this exception being thrown by Kafka.
>> >
>> > [2011-11-02 15:36:40,947] ERROR error when processing request
>> > topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
>> > (kafka.server.KafkaRequestHandlers)
>> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out of
>> > range
>> > at kafka.log.Log$.findRange(Log.scala:47)
>> > at kafka.log.Log.read(Log.scala:223)
>> > at
>> >
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
>> > at
>> >
>> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
>> > at
>> >
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
>> > at
>> >
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
>> > at kafka.network.Processor.handle(SocketServer.scala:268)
>> > at kafka.network.Processor.read(SocketServer.scala:291)
>> > at kafka.network.Processor.run(SocketServer.scala:202)
>> > at java.lang.Thread.run(Thread.java:619)
>> >
>> >
>> >
>> > I have attached the log file for the topic.
>> >
>> > Is there a way to work around this issue?
>> >
>> >
>> > Thanks,
>> > Nalin
>> >
>>
>
Re: Topics getting corrupted
Posted by Jun Rao <ju...@gmail.com>.
bin/kafka-run-class.sh kafka.tools.DumpLogSegments logfilename -noprint
Jun
On Thu, Nov 3, 2011 at 11:33 AM, Nalin DeZoysa <nd...@tagged.com> wrote:
> Could you tell me how to run DumpLogSegment?
>
>
> Thanks,
> Nalin
>
> On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > I don't see the attached log file. I guess the Apache mailing list
> doesn't
> > really allow attachments. Could you run the util DumpLogSegement? It will
> > tell you whether the log file is corrupted or not.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com>
> wrote:
> >
> > > Hi,
> > >
> > > I'm seeing that certain topics get corrupted - I can publish to it, but
> > > when I try and subscribe to a topic, the response that comes back has
> > > header error code 1. I also see this exception being thrown by Kafka.
> > >
> > > [2011-11-02 15:36:40,947] ERROR error when processing request
> > > topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
> > > (kafka.server.KafkaRequestHandlers)
> > > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out
> of
> > > range
> > > at kafka.log.Log$.findRange(Log.scala:47)
> > > at kafka.log.Log.read(Log.scala:223)
> > > at
> > >
> >
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> > > at
> > >
> >
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
> > > at
> > >
> >
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> > > at
> > >
> >
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> > > at kafka.network.Processor.handle(SocketServer.scala:268)
> > > at kafka.network.Processor.read(SocketServer.scala:291)
> > > at kafka.network.Processor.run(SocketServer.scala:202)
> > > at java.lang.Thread.run(Thread.java:619)
> > >
> > >
> > >
> > > I have attached the log file for the topic.
> > >
> > > Is there a way to work around this issue?
> > >
> > >
> > > Thanks,
> > > Nalin
> > >
> >
>
Re: Topics getting corrupted
Posted by Nalin DeZoysa <nd...@tagged.com>.
Could you tell me how to run DumpLogSegment?
Thanks,
Nalin
On Wed, Nov 2, 2011 at 6:19 PM, Jun Rao <ju...@gmail.com> wrote:
> I don't see the attached log file. I guess the Apache mailing list doesn't
> really allow attachments. Could you run the util DumpLogSegement? It will
> tell you whether the log file is corrupted or not.
>
> Thanks,
>
> Jun
>
> On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com> wrote:
>
> > Hi,
> >
> > I'm seeing that certain topics get corrupted - I can publish to it, but
> > when I try and subscribe to a topic, the response that comes back has
> > header error code 1. I also see this exception being thrown by Kafka.
> >
> > [2011-11-02 15:36:40,947] ERROR error when processing request
> > topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
> > (kafka.server.KafkaRequestHandlers)
> > kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out of
> > range
> > at kafka.log.Log$.findRange(Log.scala:47)
> > at kafka.log.Log.read(Log.scala:223)
> > at
> >
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> > at
> >
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
> > at
> >
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> > at
> >
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> > at kafka.network.Processor.handle(SocketServer.scala:268)
> > at kafka.network.Processor.read(SocketServer.scala:291)
> > at kafka.network.Processor.run(SocketServer.scala:202)
> > at java.lang.Thread.run(Thread.java:619)
> >
> >
> >
> > I have attached the log file for the topic.
> >
> > Is there a way to work around this issue?
> >
> >
> > Thanks,
> > Nalin
> >
>
Re: Topics getting corrupted
Posted by Jun Rao <ju...@gmail.com>.
I don't see the attached log file. I guess the Apache mailing list doesn't
really allow attachments. Could you run the util DumpLogSegement? It will
tell you whether the log file is corrupted or not.
Thanks,
Jun
On Wed, Nov 2, 2011 at 3:54 PM, Nalin DeZoysa <nd...@tagged.com> wrote:
> Hi,
>
> I'm seeing that certain topics get corrupted - I can publish to it, but
> when I try and subscribe to a topic, the response that comes back has
> header error code 1. I also see this exception being thrown by Kafka.
>
> [2011-11-02 15:36:40,947] ERROR error when processing request
> topic:cards_card_251791, part:0 offset:281474976724790 maxSize:1048576
> (kafka.server.KafkaRequestHandlers)
> kafka.common.OffsetOutOfRangeException: offset 281474976724790 is out of
> range
> at kafka.log.Log$.findRange(Log.scala:47)
> at kafka.log.Log.read(Log.scala:223)
> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> at
> kafka.server.KafkaRequestHandlers.handleFetchRequest(KafkaRequestHandlers.scala:107)
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$2.apply(KafkaRequestHandlers.scala:42)
> at kafka.network.Processor.handle(SocketServer.scala:268)
> at kafka.network.Processor.read(SocketServer.scala:291)
> at kafka.network.Processor.run(SocketServer.scala:202)
> at java.lang.Thread.run(Thread.java:619)
>
>
>
> I have attached the log file for the topic.
>
> Is there a way to work around this issue?
>
>
> Thanks,
> Nalin
>