Warning logs: Mixing offset types

[Anders Engström <ep...@gmail.com>]

Hi!

We're running Kafka 3.3 (at Aiven) and are seeing warning logs about offset
commits from both consumers and producers.
This happens for two of our consumers (out of hundreds). I've tried
Googling for the error (of course) but I haven't found anything obvious.

[kafka-01-78]2023-06-05T06:44:12.536334[kafka][2023-06-05 06:44:12,536]
> WARN [GroupMetadataManager brokerId=78] group: <group.id> with leader:
> <consumer-group-leader-id> has received offset commits from consumers as
> well as transactional producers. Mixing both types of offset commits will
> generally result in surprises and should be avoided.
> (kafka.coordinator.group.GroupMetadataManager:70)


Any recommendations on what to look for in the application
consumer/producer code? The applications using the consumer group id is not
really special compared to all other consumer applications.

Best Regards /Anders

Re: Warning logs: Mixing offset types

[Anders Engström <ep...@gmail.com>]

Hi, thanks for the reply! See comments below -

On Mon, Jun 5, 2023 at 9:59 AM Richard Bosch <ri...@axual.com>
wrote:

> This message indicates that the transactional producer uses the send
> offsets to transactions option.
>

We use, by default, transactional producers (setting a transaction.id). And
most services are setup to also use transactional consumers (read_committed
with auto-commit). This seems to work fine as a general pattern - and it's
only for these two consumers we see this error.


> It means that either another producer application is writing offsets
> for the consumer group as well or that you might be having a consumer
> and producer that adds offset to the transaction in the same
> application.
>
>
I was under the impression that producers store offsets in a the
__transaction_state topic, and that consumers store offsets in
__consumer_offsets. And that transactional markers are written to the
target partition by the transactional producer, and that the broker uses
this to dish out messages to consumers (transactional or non-transactional)
in a more or less transparent way.

How would the producer need to be configured in order for it to "mess with"
the consumer group id used by the consumer in this case? 🤔

Now, looking closer at the services that uses these consumer group ids,
they both use `spring-kafka` (as do all other services) with an error
handler setup (`org.springframework.kafka.listener.DefaultErrorHandler`).
This handler seems to be doing seeks on the consumer, and I'm thinking this
might be what's somehow causing this...

I'll try to dig deeper into that error handler -- but if anyone have
experience from Spring-Kafka and error handlers, feel free to jump in :)

Cheers /Anders


>
> On Mon, Jun 5, 2023 at 8:49 AM Anders Engström <ep...@gmail.com>
> wrote:
> >
> > Hi!
> >
> > We're running Kafka 3.3 (at Aiven) and are seeing warning logs about
> offset
> > commits from both consumers and producers.
> > This happens for two of our consumers (out of hundreds). I've tried
> > Googling for the error (of course) but I haven't found anything obvious.
> >
> > [kafka-01-78]2023-06-05T06:44:12.536334[kafka][2023-06-05 06:44:12,536]
> > > WARN [GroupMetadataManager brokerId=78] group: <group.id> with leader:
> > > <consumer-group-leader-id> has received offset commits from consumers
> as
> > > well as transactional producers. Mixing both types of offset commits
> will
> > > generally result in surprises and should be avoided.
> > > (kafka.coordinator.group.GroupMetadataManager:70)
> >
> >
> > Any recommendations on what to look for in the application
> > consumer/producer code? The applications using the consumer group id is
> not
> > really special compared to all other consumer applications.
> >
> > Best Regards /Anders
>

Re: Warning logs: Mixing offset types

[Richard Bosch <ri...@axual.com>]

Hi Anders,

This message indicates that the transactional producer uses the send
offsets to transactions option.
It means that either another producer application is writing offsets
for the consumer group as well or that you might be having a consumer
and producer that adds offset to the transaction in the same
application.

If the goal is to only commit offsets after processing the consumed
records then you must not commit offsets with the consumer, and
disable the auto commit offset configurations as well.
The consumer must be configured to disable automatic offset commits,
it is enabled in the Java client by default.
There might not be an explicit consumer commitSync/commitAsync call in
the consumer code, but it would be sending offset commits.

I hope this helps you to find the problem

Kind regards,

Richard Bosch
Developer Advocate
Axual BV
https://axual.com/

Kind regards,


Richard Bosch

Developer Advocate

Axual BV

E : richard.bosch@axual.com
M : +31 6 11 850 846
W : www.axual.com



On Mon, Jun 5, 2023 at 8:49 AM Anders Engström <ep...@gmail.com> wrote:
>
> Hi!
>
> We're running Kafka 3.3 (at Aiven) and are seeing warning logs about offset
> commits from both consumers and producers.
> This happens for two of our consumers (out of hundreds). I've tried
> Googling for the error (of course) but I haven't found anything obvious.
>
> [kafka-01-78]2023-06-05T06:44:12.536334[kafka][2023-06-05 06:44:12,536]
> > WARN [GroupMetadataManager brokerId=78] group: <group.id> with leader:
> > <consumer-group-leader-id> has received offset commits from consumers as
> > well as transactional producers. Mixing both types of offset commits will
> > generally result in surprises and should be avoided.
> > (kafka.coordinator.group.GroupMetadataManager:70)
>
>
> Any recommendations on what to look for in the application
> consumer/producer code? The applications using the consumer group id is not
> really special compared to all other consumer applications.
>
> Best Regards /Anders