Any tool to easily fetch a single message or a few messages starting from a given offset then exit after fetching said count of messages?

[David Luu <ma...@gmail.com>]

Hi,

I notice the kafka-console-consumer.sh script has option to fetch a max #
of messages, which can be 1 or more, and then exit. Which is nice. But as a
high level consumer, it's missing option to fetch from given offset other
than earliest & latest offsets.

Is there any off the shelf tool (CLI, simple script, etc.) that I can use
to fetch 1 or more messages from a given offset of some topic as a
consumer, then exit? Or do I have to write my own tool to do this from any
of the kafka libraries in various languages? If the later, is there any
where I can whip one up in fewest lines possible? I don't want to have to
read up and spend time crafting up a tool for this if I don't have to.

Wish this feature was already built in to the kafka toolchain.

Regards,
David

Re: Any tool to easily fetch a single message or a few messages starting from a given offset then exit after fetching said count of messages?

[Ewen Cheslack-Postava <ew...@confluent.io>]

You can also use the simple consumer shell that ships with Kafka:

bin/kafka-simple-consumer-shell.sh --broker-list mybroker:9092 --topic
mytopic --partition mypartition --offset myoffset --max-messages msgcount

-Ewen

On Fri, Jul 24, 2015 at 11:52 PM, Magnus Edenhill <ma...@edenhill.se>
wrote:

> Hi David,
>
> kafkacat does this with ease:
>   kafkacat -b mybroker -t mytopic -p mypartition -o myoffset -c msgcount
>
> If you are to pass the output to another program/script you may want to
> look in to
> the -f formatting option or -J for JSON.
>
> See here:
> https://github.com/edenhill/kafkacat
>
>
> Regards,
> Magnus
>
>
> 2015-07-25 2:20 GMT+02:00 David Luu <ma...@gmail.com>:
>
> > Hi,
> >
> > I notice the kafka-console-consumer.sh script has option to fetch a max #
> > of messages, which can be 1 or more, and then exit. Which is nice. But
> as a
> > high level consumer, it's missing option to fetch from given offset other
> > than earliest & latest offsets.
> >
> > Is there any off the shelf tool (CLI, simple script, etc.) that I can use
> > to fetch 1 or more messages from a given offset of some topic as a
> > consumer, then exit? Or do I have to write my own tool to do this from
> any
> > of the kafka libraries in various languages? If the later, is there any
> > where I can whip one up in fewest lines possible? I don't want to have to
> > read up and spend time crafting up a tool for this if I don't have to.
> >
> > Wish this feature was already built in to the kafka toolchain.
> >
> > Regards,
> > David
> >
>



-- 
Thanks,
Ewen

Re: Any tool to easily fetch a single message or a few messages starting from a given offset then exit after fetching said count of messages?

[Magnus Edenhill <ma...@edenhill.se>]

Hi David,

kafkacat does this with ease:
  kafkacat -b mybroker -t mytopic -p mypartition -o myoffset -c msgcount

If you are to pass the output to another program/script you may want to
look in to
the -f formatting option or -J for JSON.

See here:
https://github.com/edenhill/kafkacat


Regards,
Magnus


2015-07-25 2:20 GMT+02:00 David Luu <ma...@gmail.com>:

> Hi,
>
> I notice the kafka-console-consumer.sh script has option to fetch a max #
> of messages, which can be 1 or more, and then exit. Which is nice. But as a
> high level consumer, it's missing option to fetch from given offset other
> than earliest & latest offsets.
>
> Is there any off the shelf tool (CLI, simple script, etc.) that I can use
> to fetch 1 or more messages from a given offset of some topic as a
> consumer, then exit? Or do I have to write my own tool to do this from any
> of the kafka libraries in various languages? If the later, is there any
> where I can whip one up in fewest lines possible? I don't want to have to
> read up and spend time crafting up a tool for this if I don't have to.
>
> Wish this feature was already built in to the kafka toolchain.
>
> Regards,
> David
>