Consume Kafka messages in a batch

[Mark Nuttall <mk...@gmail.com>]

I know consuming Kafka messages in a batch is not currently supported. I
googled and I didn't find any real options.  I looked at hacking the Camel
classes to implement it and decided that it was too risky.

So for now we are falling back to using a Spring Kafka Consumer. The issue
with that is I still want to use Camel because we need to do some complex
EIP processing and republishing of messages. And since Spring does not know
about Camel Context startup i have to do a hack to make it wait if the
CamelContext is not in the "started" state.

Questions:

   - Is it possible to use the Spring Kafka Consumer in the "From" and be
   managed by Camel (I think not but i figured i would ask).
   - Is implementing batch consumers in the Kafka component planned/in
   progress?
   - If none of these - any ideas to make it work with the Kafka component?

Re: Consume Kafka messages in a batch

[Mark Nuttall <mk...@gmail.com>]

Thanks Otavio!

Option 1 won't work because the Camel code, while it does consume a batch
based on those settings and iterates over the enumeration, it only ever
passes one at a time. I thought about hacking that class but when we update
camel versions, i'd have to remember i did that.  :) . We cannot use the
Aggregate EIP because the only real choice is in-memory and that has the
potential for data loss. Using an rdbms for the Aggregation does not work
in k8s and additionally we are trying to bulk update a db anyway.  Also,
the offset is not passed when, auto commit is on or off (forget which one)
and the correct combination to support aggregation, the value is not passed.

I will look at 2 again. I had dug in but it seemed to be pretty
complicated.  I did dig into the Spring code to see how they did it, but
that was complex and I ran out of time.

Mainly I didnt want to duplicate effort if anyone had started this. I know
you all are slammed.



On Wed, Mar 22, 2023 at 5:24 AM Otavio Rodolfo Piske <an...@gmail.com>
wrote:

> Hi,
>
> Some ideas worth investigating:
>
> 1. Using a mix of current kafka client options (i.e.: like maxPollrecords,
> maxPollInterval, etc) along with aggregate EIP
> 2. Create your own KafkaClientFactory that wraps a custom Producer/Consumer
> wrapping the Spring Kafka Consumer.
>
> If none of this works, then I suggest opening a ticket with your
> suggestion. We have been pretty busy / overloaded with the work on Camel 4,
> so it's pretty easy for us to miss bug reports and interesting feature
> suggestions - as yours.If possible, try to provide a pseudo-code, a
> reproducer or a unit test that the community can look at and work with.
>
>
> Thanks
>
> On Wed, Mar 15, 2023 at 1:31 PM Mark Nuttall <mk...@gmail.com> wrote:
>
> > I know consuming Kafka messages in a batch is not currently supported. I
> > googled and I didn't find any real options.  I looked at hacking the
> Camel
> > classes to implement it and decided that it was too risky.
> >
> > So for now we are falling back to using a Spring Kafka Consumer. The
> issue
> > with that is I still want to use Camel because we need to do some complex
> > EIP processing and republishing of messages. And since Spring does not
> know
> > about Camel Context startup i have to do a hack to make it wait if the
> > CamelContext is not in the "started" state.
> >
> > Questions:
> >
> >    - Is it possible to use the Spring Kafka Consumer in the "From" and be
> >    managed by Camel (I think not but i figured i would ask).
> >    - Is implementing batch consumers in the Kafka component planned/in
> >    progress?
> >    - If none of these - any ideas to make it work with the Kafka
> component?
> >
>
>
> --
> Otavio R. Piske
> http://orpiske.net
>

Re: Consume Kafka messages in a batch

[Otavio Rodolfo Piske <an...@gmail.com>]

Hi,

Some ideas worth investigating:

1. Using a mix of current kafka client options (i.e.: like maxPollrecords,
maxPollInterval, etc) along with aggregate EIP
2. Create your own KafkaClientFactory that wraps a custom Producer/Consumer
wrapping the Spring Kafka Consumer.

If none of this works, then I suggest opening a ticket with your
suggestion. We have been pretty busy / overloaded with the work on Camel 4,
so it's pretty easy for us to miss bug reports and interesting feature
suggestions - as yours.If possible, try to provide a pseudo-code, a
reproducer or a unit test that the community can look at and work with.


Thanks

On Wed, Mar 15, 2023 at 1:31 PM Mark Nuttall <mk...@gmail.com> wrote:

> I know consuming Kafka messages in a batch is not currently supported. I
> googled and I didn't find any real options.  I looked at hacking the Camel
> classes to implement it and decided that it was too risky.
>
> So for now we are falling back to using a Spring Kafka Consumer. The issue
> with that is I still want to use Camel because we need to do some complex
> EIP processing and republishing of messages. And since Spring does not know
> about Camel Context startup i have to do a hack to make it wait if the
> CamelContext is not in the "started" state.
>
> Questions:
>
>    - Is it possible to use the Spring Kafka Consumer in the "From" and be
>    managed by Camel (I think not but i figured i would ask).
>    - Is implementing batch consumers in the Kafka component planned/in
>    progress?
>    - If none of these - any ideas to make it work with the Kafka component?
>


-- 
Otavio R. Piske
http://orpiske.net