Better strategy for sending a message to multiple topics

["Abel ." <ab...@amtech.mx>]

Hi,

I have this scenario where I need to send a message to multiple topics. I
create a single KafkaProducer, prepare the payload and then I call the send
method of the producer for each topic with the correspoding ProducerRecord
for the topic and the fixed message. However, I have noticed that this
procedure takes some time depending on the number of topics. For instance,
to send a message to 30 topics it takes more than 3s because each request
takes about 100ms to return from the send method. Is there a better way to
accomplish this same task? Any recommendation?

Regards,

Abel.

Re: Better strategy for sending a message to multiple topics

[Jens Rantil <je...@tink.se>]

Hi,


Why don't your consumers instead subscribe to a single topic used to broadcast to all of them? That way your consumers and producer will be much simpler.




Cheers,

Jens





–
Skickat från Mailbox

On Fri, Dec 18, 2015 at 4:16 PM, Abel . <ab...@amtech.mx> wrote:

> Hi,
> I have this scenario where I need to send a message to multiple topics. I
> create a single KafkaProducer, prepare the payload and then I call the send
> method of the producer for each topic with the correspoding ProducerRecord
> for the topic and the fixed message. However, I have noticed that this
> procedure takes some time depending on the number of topics. For instance,
> to send a message to 30 topics it takes more than 3s because each request
> takes about 100ms to return from the send method. Is there a better way to
> accomplish this same task? Any recommendation?
> Regards,
> Abel.

Re: Better strategy for sending a message to multiple topics

["Abel ." <ab...@amtech.mx>]

Hi,

Jens, I would do it that way but different queues contain different
contents. That's why I need more than a single queue to solve my problem.
However, I have a hierarchical structure of queues (like a tree or forest)
and when you push a message to a queue I need to make it available to all
the ancestors in the tree of topics. That's why I need to push to multiple
queues. It's like the most straightforward approach but I haven't found
anything better.

Dong, I think I'm using asynchronous producing but I'm not completely sure.
I'm using an instance of the KafkaProducer object and calling the send
method which returns a Future object and I'm not calling the get method of
that Future instance so I'm assuming that's asynchronous, isn't it?

About the ACK I'm not setting it so I suppose that I'm using the default
which seems to be 1. Am I right?

Thanks both for your answers.

Cheers,
Abel.

On Sun, Dec 20, 2015 at 10:35 PM, Dong Lin <li...@gmail.com> wrote:

> Did you try asynchronous produce? That should provide better performance
> because asynchronous produce doesn't block waiting for response.
>
> BTW, what is your ack setting for producer?
>
> Dong
>
> On Fri, Dec 18, 2015 at 1:16 PM, Abel . <ab...@amtech.mx> wrote:
>
> > Hi,
> >
> > I have this scenario where I need to send a message to multiple topics. I
> > create a single KafkaProducer, prepare the payload and then I call the
> send
> > method of the producer for each topic with the correspoding
> ProducerRecord
> > for the topic and the fixed message. However, I have noticed that this
> > procedure takes some time depending on the number of topics. For
> instance,
> > to send a message to 30 topics it takes more than 3s because each request
> > takes about 100ms to return from the send method. Is there a better way
> to
> > accomplish this same task? Any recommendation?
> >
> > Regards,
> >
> > Abel.
> >
>

Re: Better strategy for sending a message to multiple topics

[Dong Lin <li...@gmail.com>]

Did you try asynchronous produce? That should provide better performance
because asynchronous produce doesn't block waiting for response.

BTW, what is your ack setting for producer?

Dong

On Fri, Dec 18, 2015 at 1:16 PM, Abel . <ab...@amtech.mx> wrote:

> Hi,
>
> I have this scenario where I need to send a message to multiple topics. I
> create a single KafkaProducer, prepare the payload and then I call the send
> method of the producer for each topic with the correspoding ProducerRecord
> for the topic and the fixed message. However, I have noticed that this
> procedure takes some time depending on the number of topics. For instance,
> to send a message to 30 topics it takes more than 3s because each request
> takes about 100ms to return from the send method. Is there a better way to
> accomplish this same task? Any recommendation?
>
> Regards,
>
> Abel.
>