You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Manu Zhang <ow...@gmail.com> on 2016/01/19 13:41:37 UTC
Broker Exception: Attempt to read with a maximum offset less than
start offset
Hi all,
Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less
than start offset <https://issues.apache.org/jira/browse/KAFKA-725> still
valid ? We are seeing a similar issue when we are carrying out the yahoo's
streaming-benchmarks <https://github.com/yahoo/streaming-benchmarks> on a
4-node cluster. Our issue id is
https://github.com/gearpump/gearpump/issues/1872.
We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4 nodes
with Zookeeper on 3 of them. On each node, 4 producers produce data to a
Kafka topic with 4 partitions and 1 replica. Each producer has a throughput
of 17K messages/s. 4 consumers are distributed (not necessarily evenly)
across the cluster and consume from Kafka as fast as possible.
I tried logging the produced offsets (with callback in send) and found that
the "start offset" already existed when the consumer failed with the fetch
exception.
This happened only when producers are producing at high throughput.
Any ideas would be much appreciated.
Thanks,
Manu Zhang
Re: Broker Exception: Attempt to read with a maximum offset less than
start offset
Posted by Manu Zhang <ow...@gmail.com>.
I can reproduce this on 0.9.0.0. The error log is
[2016-01-28 16:12:32,840] ERROR [Replica Manager on Broker 1]: Error
processing fetch operation on partition [ad-events,1] offset 75510318
(kafka.server.ReplicaManager)
I also print the sent offset from producer
time partition offset
16:12:32.840 1 75510318
It seems the offset is produced and consumed at the same time.
On Wed, Jan 27, 2016 at 10:18 PM, Robert Metzger <rm...@apache.org>
wrote:
> Yes, I've asked the user to test with the 0.9.0.0 release (I saw Gwen's
> comment in KAFKA-725).
>
> I have a potentially related question: Is it an issue that both Flink and
> Gearpump* are not committing their offsets through the SimpleConsumer API?
> Flink is directly committing the offsets into ZK (and maintaining them
> internally).
> I'm asking because I see some log statements like this:
> WARN Partition [WordCount,22] on broker 0: No checkpointed highwatermark is
> found for partition [WordCount,22] (kafka.cluster.Partition)
>
> Reading from the Kafka server's code, it doesn't seem that the
> commitOffsets() request updates any internal state (at least for
> version=0).
>
> *I didn't find any call to commitOffsets() in their code.
>
>
> On Wed, Jan 27, 2016 at 2:32 PM, Ismael Juma <ml...@juma.me.uk> wrote:
>
> > Hi Manu and Robert,
> >
> > It would help to know if this still happens in trunk or the 0.9.0 branch.
> >
> > Ismael
> > On 27 Jan 2016 13:05, "Robert Metzger" <rm...@apache.org> wrote:
> >
> > > Hi Manu,
> > >
> > > in the streaming-benchmark, are seeing the issue only when reading with
> > > Gearpump, or is it triggered by a different processing framework as
> well?
> > >
> > > I'm asking because there is a Flink user who is using Kafka 0.8.2.1 as
> > well
> > > who's reporting a very similar issue on SO:
> > >
> > >
> >
> http://stackoverflow.com/questions/34982483/flink-streaming-job-switched-to-failed-status/34987963
> > > .
> > > His issue is also only present under load.
> > >
> > >
> > >
> > >
> > > On Thu, Jan 21, 2016 at 2:28 AM, Manu Zhang <ow...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Any suggestions for this issue or do I need to provide more
> > information ?
> > > > Any links I can refer to would be also very helpful.
> > > >
> > > > Thanks,
> > > > Manu Zhang
> > > >
> > > >
> > > > On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang <owenzhang1990@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Is KAFKA-725 Broker Exception: Attempt to read with a maximum
> offset
> > > less
> > > > > than start offset <https://issues.apache.org/jira/browse/KAFKA-725
> >
> > > > still
> > > > > valid ? We are seeing a similar issue when we are carrying out the
> > > > yahoo's
> > > > > streaming-benchmarks <
> https://github.com/yahoo/streaming-benchmarks>
> > > on
> > > > a
> > > > > 4-node cluster. Our issue id is
> > > > > https://github.com/gearpump/gearpump/issues/1872.
> > > > >
> > > > > We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4
> > > nodes
> > > > > with Zookeeper on 3 of them. On each node, 4 producers produce data
> > to
> > > a
> > > > > Kafka topic with 4 partitions and 1 replica. Each producer has a
> > > > throughput
> > > > > of 17K messages/s. 4 consumers are distributed (not necessarily
> > evenly)
> > > > > across the cluster and consume from Kafka as fast as possible.
> > > > >
> > > > > I tried logging the produced offsets (with callback in send) and
> > found
> > > > > that the "start offset" already existed when the consumer failed
> with
> > > the
> > > > > fetch exception.
> > > > >
> > > > > This happened only when producers are producing at high throughput.
> > > > >
> > > > > Any ideas would be much appreciated.
> > > > >
> > > > > Thanks,
> > > > > Manu Zhang
> > > > >
> > > >
> > >
> >
>
Re: Broker Exception: Attempt to read with a maximum offset less than
start offset
Posted by Robert Metzger <rm...@apache.org>.
Yes, I've asked the user to test with the 0.9.0.0 release (I saw Gwen's
comment in KAFKA-725).
I have a potentially related question: Is it an issue that both Flink and
Gearpump* are not committing their offsets through the SimpleConsumer API?
Flink is directly committing the offsets into ZK (and maintaining them
internally).
I'm asking because I see some log statements like this:
WARN Partition [WordCount,22] on broker 0: No checkpointed highwatermark is
found for partition [WordCount,22] (kafka.cluster.Partition)
Reading from the Kafka server's code, it doesn't seem that the
commitOffsets() request updates any internal state (at least for version=0).
*I didn't find any call to commitOffsets() in their code.
On Wed, Jan 27, 2016 at 2:32 PM, Ismael Juma <ml...@juma.me.uk> wrote:
> Hi Manu and Robert,
>
> It would help to know if this still happens in trunk or the 0.9.0 branch.
>
> Ismael
> On 27 Jan 2016 13:05, "Robert Metzger" <rm...@apache.org> wrote:
>
> > Hi Manu,
> >
> > in the streaming-benchmark, are seeing the issue only when reading with
> > Gearpump, or is it triggered by a different processing framework as well?
> >
> > I'm asking because there is a Flink user who is using Kafka 0.8.2.1 as
> well
> > who's reporting a very similar issue on SO:
> >
> >
> http://stackoverflow.com/questions/34982483/flink-streaming-job-switched-to-failed-status/34987963
> > .
> > His issue is also only present under load.
> >
> >
> >
> >
> > On Thu, Jan 21, 2016 at 2:28 AM, Manu Zhang <ow...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Any suggestions for this issue or do I need to provide more
> information ?
> > > Any links I can refer to would be also very helpful.
> > >
> > > Thanks,
> > > Manu Zhang
> > >
> > >
> > > On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang <ow...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset
> > less
> > > > than start offset <https://issues.apache.org/jira/browse/KAFKA-725>
> > > still
> > > > valid ? We are seeing a similar issue when we are carrying out the
> > > yahoo's
> > > > streaming-benchmarks <https://github.com/yahoo/streaming-benchmarks>
> > on
> > > a
> > > > 4-node cluster. Our issue id is
> > > > https://github.com/gearpump/gearpump/issues/1872.
> > > >
> > > > We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4
> > nodes
> > > > with Zookeeper on 3 of them. On each node, 4 producers produce data
> to
> > a
> > > > Kafka topic with 4 partitions and 1 replica. Each producer has a
> > > throughput
> > > > of 17K messages/s. 4 consumers are distributed (not necessarily
> evenly)
> > > > across the cluster and consume from Kafka as fast as possible.
> > > >
> > > > I tried logging the produced offsets (with callback in send) and
> found
> > > > that the "start offset" already existed when the consumer failed with
> > the
> > > > fetch exception.
> > > >
> > > > This happened only when producers are producing at high throughput.
> > > >
> > > > Any ideas would be much appreciated.
> > > >
> > > > Thanks,
> > > > Manu Zhang
> > > >
> > >
> >
>
Re: Broker Exception: Attempt to read with a maximum offset less than
start offset
Posted by Ismael Juma <ml...@juma.me.uk>.
Hi Manu and Robert,
It would help to know if this still happens in trunk or the 0.9.0 branch.
Ismael
On 27 Jan 2016 13:05, "Robert Metzger" <rm...@apache.org> wrote:
> Hi Manu,
>
> in the streaming-benchmark, are seeing the issue only when reading with
> Gearpump, or is it triggered by a different processing framework as well?
>
> I'm asking because there is a Flink user who is using Kafka 0.8.2.1 as well
> who's reporting a very similar issue on SO:
>
> http://stackoverflow.com/questions/34982483/flink-streaming-job-switched-to-failed-status/34987963
> .
> His issue is also only present under load.
>
>
>
>
> On Thu, Jan 21, 2016 at 2:28 AM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Any suggestions for this issue or do I need to provide more information ?
> > Any links I can refer to would be also very helpful.
> >
> > Thanks,
> > Manu Zhang
> >
> >
> > On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang <ow...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset
> less
> > > than start offset <https://issues.apache.org/jira/browse/KAFKA-725>
> > still
> > > valid ? We are seeing a similar issue when we are carrying out the
> > yahoo's
> > > streaming-benchmarks <https://github.com/yahoo/streaming-benchmarks>
> on
> > a
> > > 4-node cluster. Our issue id is
> > > https://github.com/gearpump/gearpump/issues/1872.
> > >
> > > We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4
> nodes
> > > with Zookeeper on 3 of them. On each node, 4 producers produce data to
> a
> > > Kafka topic with 4 partitions and 1 replica. Each producer has a
> > throughput
> > > of 17K messages/s. 4 consumers are distributed (not necessarily evenly)
> > > across the cluster and consume from Kafka as fast as possible.
> > >
> > > I tried logging the produced offsets (with callback in send) and found
> > > that the "start offset" already existed when the consumer failed with
> the
> > > fetch exception.
> > >
> > > This happened only when producers are producing at high throughput.
> > >
> > > Any ideas would be much appreciated.
> > >
> > > Thanks,
> > > Manu Zhang
> > >
> >
>
Re: Broker Exception: Attempt to read with a maximum offset less than
start offset
Posted by Robert Metzger <rm...@apache.org>.
Hi Manu,
in the streaming-benchmark, are seeing the issue only when reading with
Gearpump, or is it triggered by a different processing framework as well?
I'm asking because there is a Flink user who is using Kafka 0.8.2.1 as well
who's reporting a very similar issue on SO:
http://stackoverflow.com/questions/34982483/flink-streaming-job-switched-to-failed-status/34987963
.
His issue is also only present under load.
On Thu, Jan 21, 2016 at 2:28 AM, Manu Zhang <ow...@gmail.com> wrote:
> Hi,
>
> Any suggestions for this issue or do I need to provide more information ?
> Any links I can refer to would be also very helpful.
>
> Thanks,
> Manu Zhang
>
>
> On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less
> > than start offset <https://issues.apache.org/jira/browse/KAFKA-725>
> still
> > valid ? We are seeing a similar issue when we are carrying out the
> yahoo's
> > streaming-benchmarks <https://github.com/yahoo/streaming-benchmarks> on
> a
> > 4-node cluster. Our issue id is
> > https://github.com/gearpump/gearpump/issues/1872.
> >
> > We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4 nodes
> > with Zookeeper on 3 of them. On each node, 4 producers produce data to a
> > Kafka topic with 4 partitions and 1 replica. Each producer has a
> throughput
> > of 17K messages/s. 4 consumers are distributed (not necessarily evenly)
> > across the cluster and consume from Kafka as fast as possible.
> >
> > I tried logging the produced offsets (with callback in send) and found
> > that the "start offset" already existed when the consumer failed with the
> > fetch exception.
> >
> > This happened only when producers are producing at high throughput.
> >
> > Any ideas would be much appreciated.
> >
> > Thanks,
> > Manu Zhang
> >
>
Re: Broker Exception: Attempt to read with a maximum offset less than
start offset
Posted by Manu Zhang <ow...@gmail.com>.
Hi,
Any suggestions for this issue or do I need to provide more information ?
Any links I can refer to would be also very helpful.
Thanks,
Manu Zhang
On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang <ow...@gmail.com> wrote:
> Hi all,
>
> Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less
> than start offset <https://issues.apache.org/jira/browse/KAFKA-725> still
> valid ? We are seeing a similar issue when we are carrying out the yahoo's
> streaming-benchmarks <https://github.com/yahoo/streaming-benchmarks> on a
> 4-node cluster. Our issue id is
> https://github.com/gearpump/gearpump/issues/1872.
>
> We are using Kafka scala-2.10-0.8.2.1. 4 brokers are installed on 4 nodes
> with Zookeeper on 3 of them. On each node, 4 producers produce data to a
> Kafka topic with 4 partitions and 1 replica. Each producer has a throughput
> of 17K messages/s. 4 consumers are distributed (not necessarily evenly)
> across the cluster and consume from Kafka as fast as possible.
>
> I tried logging the produced offsets (with callback in send) and found
> that the "start offset" already existed when the consumer failed with the
> fetch exception.
>
> This happened only when producers are producing at high throughput.
>
> Any ideas would be much appreciated.
>
> Thanks,
> Manu Zhang
>