You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@camel.apache.org by "Claus Ibsen (Jira)" <ji...@apache.org> on 2021/06/17 13:37:00 UTC

[jira] [Commented] (CAMEL-16064) camel-kafka - Add batching consumer

    [ https://issues.apache.org/jira/browse/CAMEL-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364944#comment-17364944 ] 

Claus Ibsen commented on CAMEL-16064:
-------------------------------------

We should also look at some general batch consumer in transactional way with aggregation (something like the old camel-sjms had). And have some SPI for this, and then we can implement this for messaging systems like kafka, rabbitmq, jms, etc.

> camel-kafka - Add batching consumer
> -----------------------------------
>
>                 Key: CAMEL-16064
>                 URL: https://issues.apache.org/jira/browse/CAMEL-16064
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-kafka
>            Reporter: Claus Ibsen
>            Priority: Major
>             Fix For: 3.12.0
>
>
> It would be good to make camel-kafka and camel-vertx-kafka consumer able to batch together N messages via aggregation strategy as a single outgoing exchange. The batch should support group size, and timeout.
> Something that was in the sjms-batch component.
> The batching should use the same KafkaConsumer instance and defer the commit till the batching has been done. This requires manual offset commit by the consumer.
> The use-case has been discussed on the chat at
> https://camel.zulipchat.com/#narrow/stream/257298-camel/topic/kafka.20and.20batching



--
This message was sent by Atlassian Jira
(v8.3.4#803005)