You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "r.wiedmann@bitproquo.org" <r....@bitproquo.org> on 2022/09/19 13:15:24 UTC
Camel Kafka Component / feature request: batch producer with individual headers
Hi all,
if we look at classes KafkaProducer and KeyValueIterator, we see that it is possible to send a batch of kafka messages,
but only with identical set of headers which is determined by the „parent“ Exchange which holds the Iterator or Iterable in its body.
In our usecase, the body contains a list of Exchanges which we want to use to determine the individual headers for every batch-instance
in the same way like currently the „parent“ Exchange is used.
Actually we solve our demands by creating some copy/pasting the mentioned classes and applying minor changes to it (which is a dirty hack).
Our proposal is to define a new configuration flag named like „batchWithIndividualHeaders“ which enables the creation of individual headers.
To implement this, the constructor parameter „propagatedHeaders“ in class KeyValueHolderIterator might by replaced with some kind of
Provider-Lambda which relays to KafkaProducer::getPropagatedHeaders, but is called inside KeyValueHolderIterator for every batch-element.
IMHO this is a common usecase and worth to implement it in the camel library itself.
What do you think?
Re: Camel Kafka Component / feature request: batch producer with individual headers
Posted by re...@bitproquo.org.
PR: https://github.com/apache/camel/pull/8407
> Am 19.09.2022 um 17:55 schrieb Claus Ibsen <cl...@gmail.com>:
>
> Hi
>
> Yeah this sounds reasonable, you are welcome to create a JIRA and send a PR.
>
>
> On Mon, Sep 19, 2022 at 3:15 PM r.wiedmann@bitproquo.org <
> r.wiedmann@bitproquo.org> wrote:
>
>> Hi all,
>>
>> if we look at classes KafkaProducer and KeyValueIterator, we see that it
>> is possible to send a batch of kafka messages,
>> but only with identical set of headers which is determined by the „parent“
>> Exchange which holds the Iterator or Iterable in its body.
>>
>> In our usecase, the body contains a list of Exchanges which we want to use
>> to determine the individual headers for every batch-instance
>> in the same way like currently the „parent“ Exchange is used.
>> Actually we solve our demands by creating some copy/pasting the mentioned
>> classes and applying minor changes to it (which is a dirty hack).
>>
>>
>> Our proposal is to define a new configuration flag named like
>> „batchWithIndividualHeaders“ which enables the creation of individual
>> headers.
>>
>>
>> To implement this, the constructor parameter „propagatedHeaders“ in class
>> KeyValueHolderIterator might by replaced with some kind of
>> Provider-Lambda which relays to KafkaProducer::getPropagatedHeaders, but
>> is called inside KeyValueHolderIterator for every batch-element.
>>
>>
>> IMHO this is a common usecase and worth to implement it in the camel
>> library itself.
>>
>>
>> What do you think?
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel Kafka Component / feature request: batch producer with individual headers
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah this sounds reasonable, you are welcome to create a JIRA and send a PR.
On Mon, Sep 19, 2022 at 3:15 PM r.wiedmann@bitproquo.org <
r.wiedmann@bitproquo.org> wrote:
> Hi all,
>
> if we look at classes KafkaProducer and KeyValueIterator, we see that it
> is possible to send a batch of kafka messages,
> but only with identical set of headers which is determined by the „parent“
> Exchange which holds the Iterator or Iterable in its body.
>
> In our usecase, the body contains a list of Exchanges which we want to use
> to determine the individual headers for every batch-instance
> in the same way like currently the „parent“ Exchange is used.
> Actually we solve our demands by creating some copy/pasting the mentioned
> classes and applying minor changes to it (which is a dirty hack).
>
>
> Our proposal is to define a new configuration flag named like
> „batchWithIndividualHeaders“ which enables the creation of individual
> headers.
>
>
> To implement this, the constructor parameter „propagatedHeaders“ in class
> KeyValueHolderIterator might by replaced with some kind of
> Provider-Lambda which relays to KafkaProducer::getPropagatedHeaders, but
> is called inside KeyValueHolderIterator for every batch-element.
>
>
> IMHO this is a common usecase and worth to implement it in the camel
> library itself.
>
>
> What do you think?
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2