Message details

[Achanta Vamsi Subhash <ac...@flipkart.com>]

Hi,

We are experimenting Kafka for a MQ use-case. We found it very useful but
couldn't find the following info from the documentation:

I have a consumer logic which can say that a message consumption failed. Is
there any way I can remove the message from that partition and put it in
other topic? Can be done in an equivalent way to a database transaction,
importantly: can i do either both or none?

- How can I remove a message of a particular topic?  Is there any
client-api for it?
- How can I access a message? Is there any client-api for it?

-- 
Regards
Vamsi Subhash

Re: Message details

[Nagesh <na...@gmail.com>]

Hi,

As per AMQP standards 0.9/1.0 any messaging system for that matter is just
a pipe/pipes allows multiple producers to publish messages, and allows
multiple pointers (A pointer per group) to consume message. It is upto the
message system to discard the message on expiry.

As the message is shared between multiple consumer groups, the system
should not allow one particular consumer/group to remove the message. The
idea behind is the function of any message system is just a temporary hop
that can be shared by multiple groups.

If you want to skip the message among the group, just consume and discard
at the group level. The group specific pointer on the topic will move on
and will not be available for any other consumer in that group.

Thanks,
Nageswara Rao


On Fri, Jun 6, 2014 at 3:47 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi Achanta,
>
> Your use case is quite interesting. If I do not understand wrong you want
> to use a transaction that atomically consume on message from a partition
> and send it to another partition correct? I pre-assume by saying "remove
> the message from that partition"  you actually mean to skip consuming it in
> the future?
>
> Currently Kafka does not have functionality to remove a message from the
> log. And I am not sure what you mean by "access a message"?
>
> Guozhang
>
>
> On Thu, Jun 5, 2014 at 11:32 AM, Achanta Vamsi Subhash <
> achanta.vamsi@flipkart.com> wrote:
>
> > Hi,
> >
> > We are experimenting Kafka for a MQ use-case. We found it very useful but
> > couldn't find the following info from the documentation:
> >
> > I have a consumer logic which can say that a message consumption failed.
> Is
> > there any way I can remove the message from that partition and put it in
> > other topic? Can be done in an equivalent way to a database transaction,
> > importantly: can i do either both or none?
> >
> > - How can I remove a message of a particular topic?  Is there any
> > client-api for it?
> > - How can I access a message? Is there any client-api for it?
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
>
>
> --
> -- Guozhang
>



-- 
Thanks & Regards,
Nageswara Rao.V

Re: Message details

[Guozhang Wang <wa...@gmail.com>]

Hi Achanta,

Your use case is quite interesting. If I do not understand wrong you want
to use a transaction that atomically consume on message from a partition
and send it to another partition correct? I pre-assume by saying "remove
the message from that partition"  you actually mean to skip consuming it in
the future?

Currently Kafka does not have functionality to remove a message from the
log. And I am not sure what you mean by "access a message"?

Guozhang


On Thu, Jun 5, 2014 at 11:32 AM, Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:

> Hi,
>
> We are experimenting Kafka for a MQ use-case. We found it very useful but
> couldn't find the following info from the documentation:
>
> I have a consumer logic which can say that a message consumption failed. Is
> there any way I can remove the message from that partition and put it in
> other topic? Can be done in an equivalent way to a database transaction,
> importantly: can i do either both or none?
>
> - How can I remove a message of a particular topic?  Is there any
> client-api for it?
> - How can I access a message? Is there any client-api for it?
>
> --
> Regards
> Vamsi Subhash
>



-- 
-- Guozhang