RE: [EXTERNAL] Re: offset commit on partition revoke even if Kafka manual commit enabled

["Gv, Shivakumar" <sh...@lowes.com>]

Hi Claus,

If I choose offset repository , Is there option to use database as a shared repository.
Because in case of multiple consumers ,if  I use local repo then when the consumer joins the group it may not get assigned to same partition even if offset repo is set.

And can we still use manual commit with offset repository set?

Thanks,
Shiva

-----Original Message-----
From: Claus Ibsen <cl...@gmail.com>
Sent: Saturday, January 23, 2021 2:47 PM
To: users@camel.apache.org
Subject: [EXTERNAL] Re: offset commit on partition revoke even if Kafka manual commit enabled

*EXTERNAL SENDER*


Hi

I don't think so. So you are also "abusing" the situation a bit as you use suspend/resume in a failure situation where you don't want any offset to be stored.
In a normal use-case where you temporarily may want to suspend then the offset should be stored.

On Fri, Jan 22, 2021 at 5:05 PM Gv, Shivakumar <sh...@lowes.com> wrote:
>
> Hi Team,
>
> I am using kafka consumer with manual commit  , but in our code we have check on the target system state . if target system  is down we suspend kafka consumer and once target state is up we enable kafka consumer.
> But when we suspend /stop kafka consumer , it is forcibly committing the offset  as per below code when partitions are getting revoked. Even if manual commit is enabled.
>
> Is there a way to disable the offset commit . in this case . please help?
>
> if (partitionLastOffset != -1) {
>                 if (!endpoint.getConfiguration().isAllowManualCommit() && offsetRepository != null) {
>                     LOG.debug("Saving offset repository state {} [topic: {} partition: {} offset: {}]", threadId, topicName,
>                             partition.partition(),
>                             partitionLastOffset);
>                     offsetRepository.setState(serializeOffsetKey(partition), serializeOffsetValue(partitionLastOffset));
>                 } else if (forceCommit) {
>                     LOG.debug("Forcing commitSync {} [topic: {} partition: {} offset: {}]", threadId, topicName,
>                             partition.partition(), partitionLastOffset);
>                     consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(partitionLastOffset + 1)));
>                 }
>
> Thanks,
> shiva
>
> ________________________________
> NOTICE: All information in and attached to the e-mails below may be proprietary, confidential, privileged and otherwise protected from improper or erroneous disclosure. If you are not the sender's intended recipient, you are not authorized to intercept, read, print, retain, copy, forward, or disseminate this message. If you have erroneously received this communication, please notify the sender immediately by phone (704-758-1000) or by e-mail and destroy all copies of this message electronic, paper, or otherwise. By transmitting documents via this email: Users, Customers, Suppliers and Vendors collectively acknowledge and agree the transmittal of information via email is voluntary, is offered as a convenience, and is not a secured method of communication; Not to transmit any payment information E.G. credit card, debit card, checking account, wire transfer information, passwords, or sensitive and personal information E.G. Driver's license, DOB, social security, or any other information the user wishes to remain confidential; To transmit only non-confidential information such as plans, pictures and drawings and to assume all risk and liability for and indemnify Lowe's from any claims, losses or damages that may arise from the transmittal of documents or including non-confidential information in the body of an email transmittal. Thank you.



--
Claus Ibsen
-----------------
https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdavsclaus.com%2F&amp;data=04%7C01%7Cshivakumar.gv%40lowes.com%7C9c6ec40172354cfe9a7b08d8bf7fb20e%7Cbcfa3e87841e48c7983b584159dd1a69%7C0%7C0%7C637469902495631989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=2M0pJFXnPQoqkj1uxcZzhVZYPASX4ouId483BeUCqCY%3D&amp;reserved=0 @davsclaus Camel in Action 2: https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.manning.com%2Fibsen2&amp;data=04%7C01%7Cshivakumar.gv%40lowes.com%7C9c6ec40172354cfe9a7b08d8bf7fb20e%7Cbcfa3e87841e48c7983b584159dd1a69%7C0%7C0%7C637469902495631989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=tXJkkVog%2FAxBI5kroPz%2F4K7nTqOPvoLb339Rlr8omIg%3D&amp;reserved=0
________________________________
NOTICE: All information in and attached to the e-mails below may be proprietary, confidential, privileged and otherwise protected from improper or erroneous disclosure. If you are not the sender's intended recipient, you are not authorized to intercept, read, print, retain, copy, forward, or disseminate this message. If you have erroneously received this communication, please notify the sender immediately by phone (704-758-1000) or by e-mail and destroy all copies of this message electronic, paper, or otherwise. By transmitting documents via this email: Users, Customers, Suppliers and Vendors collectively acknowledge and agree the transmittal of information via email is voluntary, is offered as a convenience, and is not a secured method of communication; Not to transmit any payment information E.G. credit card, debit card, checking account, wire transfer information, passwords, or sensitive and personal information E.G. Driver's license, DOB, social security, or any other information the user wishes to remain confidential; To transmit only non-confidential information such as plans, pictures and drawings and to assume all risk and liability for and indemnify Lowe's from any claims, losses or damages that may arise from the transmittal of documents or including non-confidential information in the body of an email transmittal. Thank you.