You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Victor L <vl...@gmail.com> on 2015/04/15 23:42:09 UTC
offset value problem
i have 0.8.1 producer/consumer... I start producer first but when i start
consumer i see the following :
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
ConsumerService
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|main.go:47|2015/04/15 21:26:29|Influxdb client
created:11.1.75.1:8086 db=logging
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
partitions are:20
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
21:26:30|curPartitions:[3 2 1 0 5 4]
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=3 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=5 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=4 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15 21:26:30|Used[6]=false
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
partition index is:6
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
is:106948
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ consumer
initialization ready
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
partitions are:20
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
21:26:30|curPartitions:[2 1 0]
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15 21:26:30|Used[3]=false
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
partition index is:3
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
is:1065377
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ consumer
initialization ready
Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
ERROR|10cf22c161e3|1|consumer|main.go:82|2015/04/15 21:26:30|Unable to
Receive Message:kafka server: The requested offset is outside the range of
offsets maintained by the server for the given topic/partition.
The offset in zookeeper is 200, so where's Offsetvalue :1065377 coming from
and how can i reset it? I see the same problem even if I delete/re-create
topics before I try to start producer/consumer pair
Than you
Re: offset value problem
Posted by Alexander Mudryk <al...@gmail.com>.
2015-04-17 14:27 GMT+03:00 Victor L <vl...@gmail.com>:
> Thank you, it is very helpful. I am using zookeeper. Does "rmr /kafka"
> remove zookeeper record associated with topic "kafka"?
>
Yes. But anything will be good if you recreate in some way (manually or by
script) all topics after kafka restart.
> On Fri, Apr 17, 2015 at 2:50 AM, Alexander Mudryk <al...@gmail.com>
> wrote:
>
> > Hello!
> >
> > "Offsetvalue :1065377" likely is coming from Zookeeper.
> > Do you use it?
> > If so, when you delete/re-create topics you should clean data in
> Zookeeper
> > too. For example:
> >
> > run
> > user@host:~$ KAFKA_PATH/bin/zookeeper-shell.sh 127.0.0.1:2181
> > and type theese commands:
> > rmr /kafka
> > create /kafka ''
> >
> > Hope this will be helpful. Good luck!
> >
> > 2015-04-16 0:42 GMT+03:00 Victor L <vl...@gmail.com>:
> >
> > > i have 0.8.1 producer/consumer... I start producer first but when i
> start
> > > consumer i see the following :
> > >
> > > Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
> > > ConsumerService
> > > Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|main.go:47|2015/04/15 21:26:29|Influxdb
> > client
> > > created:11.1.75.1:8086 db=logging
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> > > partitions are:20
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> > > 21:26:30|curPartitions:[3 2 1 0 5 4]
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=3
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=5
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=4
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15
> > 21:26:30|Used[6]=false
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> > > partition index is:6
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15
> 21:26:30|Offsetvalue
> > > is:106948
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ
> > consumer
> > > initialization ready
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> > > partitions are:20
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> > > 21:26:30|curPartitions:[2 1 0]
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0
> true
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15
> > 21:26:30|Used[3]=false
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> > > partition index is:3
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15
> 21:26:30|Offsetvalue
> > > is:1065377
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ
> > consumer
> > > initialization ready
> > > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > > ERROR|10cf22c161e3|1|consumer|main.go:82|2015/04/15 21:26:30|Unable to
> > > Receive Message:kafka server: The requested offset is outside the range
> > of
> > > offsets maintained by the server for the given topic/partition.
> > >
> > > The offset in zookeeper is 200, so where's Offsetvalue :1065377 coming
> > from
> > > and how can i reset it? I see the same problem even if I
> delete/re-create
> > > topics before I try to start producer/consumer pair
> > >
> > > Than you
> > >
> >
> >
> >
> > --
> > С уважением,
> > Александр Мудрык.
> >
>
--
С уважением,
Александр Мудрык.
Re: offset value problem
Posted by Victor L <vl...@gmail.com>.
Thank you, it is very helpful. I am using zookeeper. Does "rmr /kafka"
remove zookeeper record associated with topic "kafka"?
On Fri, Apr 17, 2015 at 2:50 AM, Alexander Mudryk <al...@gmail.com>
wrote:
> Hello!
>
> "Offsetvalue :1065377" likely is coming from Zookeeper.
> Do you use it?
> If so, when you delete/re-create topics you should clean data in Zookeeper
> too. For example:
>
> run
> user@host:~$ KAFKA_PATH/bin/zookeeper-shell.sh 127.0.0.1:2181
> and type theese commands:
> rmr /kafka
> create /kafka ''
>
> Hope this will be helpful. Good luck!
>
> 2015-04-16 0:42 GMT+03:00 Victor L <vl...@gmail.com>:
>
> > i have 0.8.1 producer/consumer... I start producer first but when i start
> > consumer i see the following :
> >
> > Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
> > ConsumerService
> > Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|main.go:47|2015/04/15 21:26:29|Influxdb
> client
> > created:11.1.75.1:8086 db=logging
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> > partitions are:20
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> > 21:26:30|curPartitions:[3 2 1 0 5 4]
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=3 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=5 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=4 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15
> 21:26:30|Used[6]=false
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> > partition index is:6
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
> > is:106948
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ
> consumer
> > initialization ready
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> > partitions are:20
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> > 21:26:30|curPartitions:[2 1 0]
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15
> 21:26:30|Used[3]=false
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> > partition index is:3
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
> > is:1065377
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ
> consumer
> > initialization ready
> > Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> > ERROR|10cf22c161e3|1|consumer|main.go:82|2015/04/15 21:26:30|Unable to
> > Receive Message:kafka server: The requested offset is outside the range
> of
> > offsets maintained by the server for the given topic/partition.
> >
> > The offset in zookeeper is 200, so where's Offsetvalue :1065377 coming
> from
> > and how can i reset it? I see the same problem even if I delete/re-create
> > topics before I try to start producer/consumer pair
> >
> > Than you
> >
>
>
>
> --
> С уважением,
> Александр Мудрык.
>
Re: offset value problem
Posted by Alexander Mudryk <al...@gmail.com>.
Hello!
"Offsetvalue :1065377" likely is coming from Zookeeper.
Do you use it?
If so, when you delete/re-create topics you should clean data in Zookeeper
too. For example:
run
user@host:~$ KAFKA_PATH/bin/zookeeper-shell.sh 127.0.0.1:2181
and type theese commands:
rmr /kafka
create /kafka ''
Hope this will be helpful. Good luck!
2015-04-16 0:42 GMT+03:00 Victor L <vl...@gmail.com>:
> i have 0.8.1 producer/consumer... I start producer first but when i start
> consumer i see the following :
>
> Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
> ConsumerService
> Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|main.go:47|2015/04/15 21:26:29|Influxdb client
> created:11.1.75.1:8086 db=logging
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> partitions are:20
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> 21:26:30|curPartitions:[3 2 1 0 5 4]
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=3 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=5 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=4 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15 21:26:30|Used[6]=false
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> partition index is:6
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
> is:106948
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ consumer
> initialization ready
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:145|2015/04/15 21:26:30|No of
> partitions are:20
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:343|2015/04/15
> 21:26:30|curPartitions:[2 1 0]
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=2 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=1 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:354|2015/04/15 21:26:30|Used=0 true
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:359|2015/04/15 21:26:30|Used[3]=false
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:184|2015/04/15 21:26:30|Availalbe
> partition index is:3
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:221|2015/04/15 21:26:30|Offsetvalue
> is:1065377
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> INFO|10cf22c161e3|1|consumer|msgq.go:238|2015/04/15 21:26:30|msgQ consumer
> initialization ready
> Apr 15 21:26:30 node-1cc1de6d1224 docker[32360]:
> ERROR|10cf22c161e3|1|consumer|main.go:82|2015/04/15 21:26:30|Unable to
> Receive Message:kafka server: The requested offset is outside the range of
> offsets maintained by the server for the given topic/partition.
>
> The offset in zookeeper is 200, so where's Offsetvalue :1065377 coming from
> and how can i reset it? I see the same problem even if I delete/re-create
> topics before I try to start producer/consumer pair
>
> Than you
>
--
С уважением,
Александр Мудрык.