You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rahul Singh <ra...@smartsensesolutions.com> on 2019/01/21 13:13:06 UTC
How to acknowledge after consuming the message from Kafka topic?
Hi All,
I am testing kafka locally, I am able to produce and consume message. But,
after consuming the message from topic I want to acknowledge.
Looking for solution. Please revert if anyone have.
Thanks & Regards
Rahul Singh
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Rahul Singh <ra...@smartsensesolutions.com>.
Hi Daniel,
This is my code. Hopes it looks understandable, thanks :)
const kafka = require('kafka-node');
const ConsumerGroup = kafka.ConsumerGroup;
let options = {
kafkaHost: '127.0.0.1:9092',
groupId: 'DualTest',
autoCommit: false,
// autoCommitIntervalMs: 5000,
protocol: ['roundrobin'],
fromOffset: 'latest',
outOfRangeOffset: 'earliest',
sessionTimeout: 15000,
fetchMaxBytes: 10 * 1024 * 1024,
}
let consumerGroup = new ConsumerGroup(options, 'topicName')
consumerGroup.on('message', (message) => {
console.log(message);
consumerGroup.commit((err, res) => {
if (err) {
console.error(err)
} else {
console.log(res)
}
});
});
consumerGroup.on('error', (err) => {
console.log(`Error Occured ${err}`)
});
Here, the autoCommit property is set to false and committing manually by
consumerGroup.commit(), but when I restart the consumer it consumes all the
offsets from starting.
Thanks
On Mon, Jan 21, 2019 at 11:48 PM Daniel Hinojosa <
dhinojosa@evolutionnext.com> wrote:
> Show some code Rahul.
>
> On Mon, Jan 21, 2019 at 11:02 AM Rahul Singh <
> rahul.singh@smartsensesolutions.com> wrote:
>
> > I am using node-kafka, I have used consumer.commit to commit offsets but
> > don't know why when I restart the consumer it consume the committed
> > offsets.
> >
> > Thanks
> >
> > On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen <hans@confluent.io wrote:
> >
> > > Are you using kafka-node or node-rdkafka? In either case you should
> call
> > > Consumer.commit(cb) or something similar to manually commit offsets
> (aka
> > > acknowledge messages).
> > >
> > > Alternatively so can set a config parameter on the consumer to
> > autoCommit.
> > >
> > > https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
> > >
> > > https://github.com/Blizzard/node-rdkafka/blob/master/README.md
> > >
> > > -hans
> > >
> > > > On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> > > rahul.singh@smartsensesolutions.com> wrote:
> > > >
> > > > I am using in Node with node-kafka module.
> > > >
> > > >> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com>
> wrote:
> > > >>
> > > >> Please read KafkaConsumer javadoc - your answer is already there.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> > > >> rahul.singh@smartsensesolutions.com> wrote:
> > > >>
> > > >>> Hi All,
> > > >>>
> > > >>> I am testing kafka locally, I am able to produce and consume
> message.
> > > >> But,
> > > >>> after consuming the message from topic I want to acknowledge.
> > > >>>
> > > >>> Looking for solution. Please revert if anyone have.
> > > >>>
> > > >>> Thanks & Regards
> > > >>> Rahul Singh
> > > >>>
> > > >>
> > >
> >
>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Daniel Hinojosa <dh...@evolutionnext.com>.
Show some code Rahul.
On Mon, Jan 21, 2019 at 11:02 AM Rahul Singh <
rahul.singh@smartsensesolutions.com> wrote:
> I am using node-kafka, I have used consumer.commit to commit offsets but
> don't know why when I restart the consumer it consume the committed
> offsets.
>
> Thanks
>
> On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen <hans@confluent.io wrote:
>
> > Are you using kafka-node or node-rdkafka? In either case you should call
> > Consumer.commit(cb) or something similar to manually commit offsets (aka
> > acknowledge messages).
> >
> > Alternatively so can set a config parameter on the consumer to
> autoCommit.
> >
> > https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
> >
> > https://github.com/Blizzard/node-rdkafka/blob/master/README.md
> >
> > -hans
> >
> > > On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> > rahul.singh@smartsensesolutions.com> wrote:
> > >
> > > I am using in Node with node-kafka module.
> > >
> > >> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
> > >>
> > >> Please read KafkaConsumer javadoc - your answer is already there.
> > >>
> > >> Thanks,
> > >>
> > >> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> > >> rahul.singh@smartsensesolutions.com> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> I am testing kafka locally, I am able to produce and consume message.
> > >> But,
> > >>> after consuming the message from topic I want to acknowledge.
> > >>>
> > >>> Looking for solution. Please revert if anyone have.
> > >>>
> > >>> Thanks & Regards
> > >>> Rahul Singh
> > >>>
> > >>
> >
>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Rahul Singh <ra...@smartsensesolutions.com>.
Hi Hans,
Let me correct this. I am using kafka-node client not node-kafka.
On Tue, Jan 22, 2019 at 3:56 AM Hans Jespersen <ha...@confluent.io> wrote:
> Do you mean this node-kafka from 4 years ago (
> https://github.com/sutoiku/node-kafka)?
>
> If so that’s a very very old client, only supports Apache Kafka 0.8 and
> stores offsets in zookeeper (which Kafka 0.9 and above no longer do).
>
> I recommend you use a more up to date nodejs kafka client than this one.
>
> -hans
>
> > On Jan 21, 2019, at 10:02 AM, Rahul Singh <
> rahul.singh@smartsensesolutions.com> wrote:
> >
> > I am using node-kafka, I have used consumer.commit to commit offsets but
> > don't know why when I restart the consumer it consume the committed
> offsets.
> >
> > Thanks
> >
> >> On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen <hans@confluent.io wrote:
> >>
> >> Are you using kafka-node or node-rdkafka? In either case you should call
> >> Consumer.commit(cb) or something similar to manually commit offsets (aka
> >> acknowledge messages).
> >>
> >> Alternatively so can set a config parameter on the consumer to
> autoCommit.
> >>
> >> https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
> >>
> >> https://github.com/Blizzard/node-rdkafka/blob/master/README.md
> >>
> >> -hans
> >>
> >>> On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> >> rahul.singh@smartsensesolutions.com> wrote:
> >>>
> >>> I am using in Node with node-kafka module.
> >>>
> >>>> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
> >>>>
> >>>> Please read KafkaConsumer javadoc - your answer is already there.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> >>>> rahul.singh@smartsensesolutions.com> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I am testing kafka locally, I am able to produce and consume message.
> >>>> But,
> >>>>> after consuming the message from topic I want to acknowledge.
> >>>>>
> >>>>> Looking for solution. Please revert if anyone have.
> >>>>>
> >>>>> Thanks & Regards
> >>>>> Rahul Singh
> >>>>>
> >>>>
> >>
>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Hans Jespersen <ha...@confluent.io>.
Do you mean this node-kafka from 4 years ago (https://github.com/sutoiku/node-kafka)?
If so that’s a very very old client, only supports Apache Kafka 0.8 and stores offsets in zookeeper (which Kafka 0.9 and above no longer do).
I recommend you use a more up to date nodejs kafka client than this one.
-hans
> On Jan 21, 2019, at 10:02 AM, Rahul Singh <ra...@smartsensesolutions.com> wrote:
>
> I am using node-kafka, I have used consumer.commit to commit offsets but
> don't know why when I restart the consumer it consume the committed offsets.
>
> Thanks
>
>> On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen <hans@confluent.io wrote:
>>
>> Are you using kafka-node or node-rdkafka? In either case you should call
>> Consumer.commit(cb) or something similar to manually commit offsets (aka
>> acknowledge messages).
>>
>> Alternatively so can set a config parameter on the consumer to autoCommit.
>>
>> https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
>>
>> https://github.com/Blizzard/node-rdkafka/blob/master/README.md
>>
>> -hans
>>
>>> On Jan 21, 2019, at 5:17 AM, Rahul Singh <
>> rahul.singh@smartsensesolutions.com> wrote:
>>>
>>> I am using in Node with node-kafka module.
>>>
>>>> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
>>>>
>>>> Please read KafkaConsumer javadoc - your answer is already there.
>>>>
>>>> Thanks,
>>>>
>>>> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
>>>> rahul.singh@smartsensesolutions.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am testing kafka locally, I am able to produce and consume message.
>>>> But,
>>>>> after consuming the message from topic I want to acknowledge.
>>>>>
>>>>> Looking for solution. Please revert if anyone have.
>>>>>
>>>>> Thanks & Regards
>>>>> Rahul Singh
>>>>>
>>>>
>>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Rahul Singh <ra...@smartsensesolutions.com>.
I am using node-kafka, I have used consumer.commit to commit offsets but
don't know why when I restart the consumer it consume the committed offsets.
Thanks
On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen <hans@confluent.io wrote:
> Are you using kafka-node or node-rdkafka? In either case you should call
> Consumer.commit(cb) or something similar to manually commit offsets (aka
> acknowledge messages).
>
> Alternatively so can set a config parameter on the consumer to autoCommit.
>
> https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
>
> https://github.com/Blizzard/node-rdkafka/blob/master/README.md
>
> -hans
>
> > On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> rahul.singh@smartsensesolutions.com> wrote:
> >
> > I am using in Node with node-kafka module.
> >
> >> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
> >>
> >> Please read KafkaConsumer javadoc - your answer is already there.
> >>
> >> Thanks,
> >>
> >> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> >> rahul.singh@smartsensesolutions.com> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I am testing kafka locally, I am able to produce and consume message.
> >> But,
> >>> after consuming the message from topic I want to acknowledge.
> >>>
> >>> Looking for solution. Please revert if anyone have.
> >>>
> >>> Thanks & Regards
> >>> Rahul Singh
> >>>
> >>
>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Hans Jespersen <ha...@confluent.io>.
Are you using kafka-node or node-rdkafka? In either case you should call Consumer.commit(cb) or something similar to manually commit offsets (aka acknowledge messages).
Alternatively so can set a config parameter on the consumer to autoCommit.
https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
https://github.com/Blizzard/node-rdkafka/blob/master/README.md
-hans
> On Jan 21, 2019, at 5:17 AM, Rahul Singh <ra...@smartsensesolutions.com> wrote:
>
> I am using in Node with node-kafka module.
>
>> On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
>>
>> Please read KafkaConsumer javadoc - your answer is already there.
>>
>> Thanks,
>>
>> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
>> rahul.singh@smartsensesolutions.com> wrote:
>>
>>> Hi All,
>>>
>>> I am testing kafka locally, I am able to produce and consume message.
>> But,
>>> after consuming the message from topic I want to acknowledge.
>>>
>>> Looking for solution. Please revert if anyone have.
>>>
>>> Thanks & Regards
>>> Rahul Singh
>>>
>>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by Rahul Singh <ra...@smartsensesolutions.com>.
I am using in Node with node-kafka module.
On Mon, Jan 21, 2019 at 6:45 PM M. Manna <ma...@gmail.com> wrote:
> Please read KafkaConsumer javadoc - your answer is already there.
>
> Thanks,
>
> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> rahul.singh@smartsensesolutions.com> wrote:
>
> > Hi All,
> >
> > I am testing kafka locally, I am able to produce and consume message.
> But,
> > after consuming the message from topic I want to acknowledge.
> >
> > Looking for solution. Please revert if anyone have.
> >
> > Thanks & Regards
> > Rahul Singh
> >
>
Re: How to acknowledge after consuming the message from Kafka topic?
Posted by "M. Manna" <ma...@gmail.com>.
Please read KafkaConsumer javadoc - your answer is already there.
Thanks,
On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
rahul.singh@smartsensesolutions.com> wrote:
> Hi All,
>
> I am testing kafka locally, I am able to produce and consume message. But,
> after consuming the message from topic I want to acknowledge.
>
> Looking for solution. Please revert if anyone have.
>
> Thanks & Regards
> Rahul Singh
>