You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pierre Coquentin <pi...@gmail.com> on 2018/07/31 09:49:14 UTC
How to know the begin offset of a partition
Hi
How do we get the starting offset of a partition from a topic when we have
a retention policy configured?
I've checked in shell script kafka-topic.sh or kafka-console-consumer.sh
but found nothing so far. Do we have to implement our own solution to get
this information?
It could be nice in kafka-console-consumer.sh to display LOG-BEGIN-OFFSET like
it already displays LOG-END-OFFSET.
Thanks,
Pierre
Re: How to know the begin offset of a partition
Posted by R Krishna <kr...@gmail.com>.
Not the best idea but you can open another listener that is not SASL
enabled until they patch is released.
On Tue, Jul 31, 2018 at 8:17 AM, Pierre Coquentin <
pierre.coquentin@gmail.com> wrote:
> thanks, too bad :(
>
> On Tue, Jul 31, 2018 at 4:44 PM Gabriele Paggi <ga...@gmail.com>
> wrote:
>
> > Hi Pierre,
> >
> > Indeed, it doesn't support SASL_SSL.
> > The issue is being worked on here:
> > https://issues.apache.org/jira/browse/KAFKA-5235
> >
> > On Tue, Jul 31, 2018 at 2:19 PM Pierre Coquentin <
> > pierre.coquentin@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > Thanks for the answer but GetOffsetShell doesn't work with SSL and SASL
> > > enabled. It doesn't seem possible to provide the consumer.config like
> we
> > do
> > > for kafka-console-consumer.sh.
> > >
> > > On Tue, Jul 31, 2018 at 12:19 PM Gabriele Paggi <
> > gabriele.paggi@gmail.com>
> > > wrote:
> > >
> > > > Hi Pierre,
> > > >
> > > > You can use kafka.tools.GetOffsetShell, which will return the
> earliest
> > > > offset for each partition in a given topic.
> > > > If you want to see the latest available ones, change "--time -2" with
> > > > "--time -1"
> > > >
> > > > gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell
> > > > --broker-list $(hostname -f):9092 --topic logs --time -2
> > > > logs:8:268728
> > > > logs:11:268544
> > > > logs:2:269047
> > > > logs:5:268855
> > > > logs:4:268919
> > > > logs:7:268781
> > > > logs:1:269115
> > > > logs:10:268636
> > > > logs:9:268676
> > > > logs:3:268962
> > > > logs:6:268812
> > > > logs:0:269176
> > > > gpaggi@kafkalog001:~$
> > > >
> > > >
> > > > On Tue, Jul 31, 2018 at 11:49 AM Pierre Coquentin <
> > > > pierre.coquentin@gmail.com> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > How do we get the starting offset of a partition from a topic when
> we
> > > > have
> > > > > a retention policy configured?
> > > > > I've checked in shell script kafka-topic.sh or
> > > kafka-console-consumer.sh
> > > > > but found nothing so far. Do we have to implement our own solution
> to
> > > get
> > > > > this information?
> > > > > It could be nice in kafka-console-consumer.sh to display
> > > LOG-BEGIN-OFFSET
> > > > > like
> > > > > it already displays LOG-END-OFFSET.
> > > > > Thanks,
> > > > >
> > > > > Pierre
> > > > >
> > > >
> > > >
> > > > --
> > > > Gabriele
> > > >
> > >
> >
>
--
Radha Krishna, Proddaturi
253-234-5657
Re: How to know the begin offset of a partition
Posted by Pierre Coquentin <pi...@gmail.com>.
thanks, too bad :(
On Tue, Jul 31, 2018 at 4:44 PM Gabriele Paggi <ga...@gmail.com>
wrote:
> Hi Pierre,
>
> Indeed, it doesn't support SASL_SSL.
> The issue is being worked on here:
> https://issues.apache.org/jira/browse/KAFKA-5235
>
> On Tue, Jul 31, 2018 at 2:19 PM Pierre Coquentin <
> pierre.coquentin@gmail.com>
> wrote:
>
> > Hi
> >
> > Thanks for the answer but GetOffsetShell doesn't work with SSL and SASL
> > enabled. It doesn't seem possible to provide the consumer.config like we
> do
> > for kafka-console-consumer.sh.
> >
> > On Tue, Jul 31, 2018 at 12:19 PM Gabriele Paggi <
> gabriele.paggi@gmail.com>
> > wrote:
> >
> > > Hi Pierre,
> > >
> > > You can use kafka.tools.GetOffsetShell, which will return the earliest
> > > offset for each partition in a given topic.
> > > If you want to see the latest available ones, change "--time -2" with
> > > "--time -1"
> > >
> > > gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell
> > > --broker-list $(hostname -f):9092 --topic logs --time -2
> > > logs:8:268728
> > > logs:11:268544
> > > logs:2:269047
> > > logs:5:268855
> > > logs:4:268919
> > > logs:7:268781
> > > logs:1:269115
> > > logs:10:268636
> > > logs:9:268676
> > > logs:3:268962
> > > logs:6:268812
> > > logs:0:269176
> > > gpaggi@kafkalog001:~$
> > >
> > >
> > > On Tue, Jul 31, 2018 at 11:49 AM Pierre Coquentin <
> > > pierre.coquentin@gmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > How do we get the starting offset of a partition from a topic when we
> > > have
> > > > a retention policy configured?
> > > > I've checked in shell script kafka-topic.sh or
> > kafka-console-consumer.sh
> > > > but found nothing so far. Do we have to implement our own solution to
> > get
> > > > this information?
> > > > It could be nice in kafka-console-consumer.sh to display
> > LOG-BEGIN-OFFSET
> > > > like
> > > > it already displays LOG-END-OFFSET.
> > > > Thanks,
> > > >
> > > > Pierre
> > > >
> > >
> > >
> > > --
> > > Gabriele
> > >
> >
>
Re: How to know the begin offset of a partition
Posted by Gabriele Paggi <ga...@gmail.com>.
Hi Pierre,
Indeed, it doesn't support SASL_SSL.
The issue is being worked on here:
https://issues.apache.org/jira/browse/KAFKA-5235
On Tue, Jul 31, 2018 at 2:19 PM Pierre Coquentin <pi...@gmail.com>
wrote:
> Hi
>
> Thanks for the answer but GetOffsetShell doesn't work with SSL and SASL
> enabled. It doesn't seem possible to provide the consumer.config like we do
> for kafka-console-consumer.sh.
>
> On Tue, Jul 31, 2018 at 12:19 PM Gabriele Paggi <ga...@gmail.com>
> wrote:
>
> > Hi Pierre,
> >
> > You can use kafka.tools.GetOffsetShell, which will return the earliest
> > offset for each partition in a given topic.
> > If you want to see the latest available ones, change "--time -2" with
> > "--time -1"
> >
> > gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell
> > --broker-list $(hostname -f):9092 --topic logs --time -2
> > logs:8:268728
> > logs:11:268544
> > logs:2:269047
> > logs:5:268855
> > logs:4:268919
> > logs:7:268781
> > logs:1:269115
> > logs:10:268636
> > logs:9:268676
> > logs:3:268962
> > logs:6:268812
> > logs:0:269176
> > gpaggi@kafkalog001:~$
> >
> >
> > On Tue, Jul 31, 2018 at 11:49 AM Pierre Coquentin <
> > pierre.coquentin@gmail.com> wrote:
> >
> > > Hi
> > >
> > > How do we get the starting offset of a partition from a topic when we
> > have
> > > a retention policy configured?
> > > I've checked in shell script kafka-topic.sh or
> kafka-console-consumer.sh
> > > but found nothing so far. Do we have to implement our own solution to
> get
> > > this information?
> > > It could be nice in kafka-console-consumer.sh to display
> LOG-BEGIN-OFFSET
> > > like
> > > it already displays LOG-END-OFFSET.
> > > Thanks,
> > >
> > > Pierre
> > >
> >
> >
> > --
> > Gabriele
> >
>
Re: How to know the begin offset of a partition
Posted by Pierre Coquentin <pi...@gmail.com>.
Hi
Thanks for the answer but GetOffsetShell doesn't work with SSL and SASL
enabled. It doesn't seem possible to provide the consumer.config like we do
for kafka-console-consumer.sh.
On Tue, Jul 31, 2018 at 12:19 PM Gabriele Paggi <ga...@gmail.com>
wrote:
> Hi Pierre,
>
> You can use kafka.tools.GetOffsetShell, which will return the earliest
> offset for each partition in a given topic.
> If you want to see the latest available ones, change "--time -2" with
> "--time -1"
>
> gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell
> --broker-list $(hostname -f):9092 --topic logs --time -2
> logs:8:268728
> logs:11:268544
> logs:2:269047
> logs:5:268855
> logs:4:268919
> logs:7:268781
> logs:1:269115
> logs:10:268636
> logs:9:268676
> logs:3:268962
> logs:6:268812
> logs:0:269176
> gpaggi@kafkalog001:~$
>
>
> On Tue, Jul 31, 2018 at 11:49 AM Pierre Coquentin <
> pierre.coquentin@gmail.com> wrote:
>
> > Hi
> >
> > How do we get the starting offset of a partition from a topic when we
> have
> > a retention policy configured?
> > I've checked in shell script kafka-topic.sh or kafka-console-consumer.sh
> > but found nothing so far. Do we have to implement our own solution to get
> > this information?
> > It could be nice in kafka-console-consumer.sh to display LOG-BEGIN-OFFSET
> > like
> > it already displays LOG-END-OFFSET.
> > Thanks,
> >
> > Pierre
> >
>
>
> --
> Gabriele
>
Re: How to know the begin offset of a partition
Posted by Gabriele Paggi <ga...@gmail.com>.
Hi Pierre,
You can use kafka.tools.GetOffsetShell, which will return the earliest
offset for each partition in a given topic.
If you want to see the latest available ones, change "--time -2" with
"--time -1"
gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list $(hostname -f):9092 --topic logs --time -2
logs:8:268728
logs:11:268544
logs:2:269047
logs:5:268855
logs:4:268919
logs:7:268781
logs:1:269115
logs:10:268636
logs:9:268676
logs:3:268962
logs:6:268812
logs:0:269176
gpaggi@kafkalog001:~$
On Tue, Jul 31, 2018 at 11:49 AM Pierre Coquentin <
pierre.coquentin@gmail.com> wrote:
> Hi
>
> How do we get the starting offset of a partition from a topic when we have
> a retention policy configured?
> I've checked in shell script kafka-topic.sh or kafka-console-consumer.sh
> but found nothing so far. Do we have to implement our own solution to get
> this information?
> It could be nice in kafka-console-consumer.sh to display LOG-BEGIN-OFFSET
> like
> it already displays LOG-END-OFFSET.
> Thanks,
>
> Pierre
>
--
Gabriele