[GitHub] [pulsar] lhotari commented on issue #12022: How to dynamically produce and consume topic data

[GitBox <gi...@apache.org>]

lhotari commented on issue #12022:
URL: https://github.com/apache/pulsar/issues/12022#issuecomment-919862391


   > When producing data, I want to dynamically send it to different topics. The topics may be in different namespaces and different tenants.
   
   In Pulsar client, you must create a separate producer for sending to a different topic. There are many ways to achieve what you are describing. The simplest approach is to create a producer, send the message and then close the producer. However, for most cases, this wouldn't be efficient, and producer caching is needed.
   This challenge is handled in the [Reactive Streams adapter for Pulsar Java Client](https://github.com/lhotari/reactive-pulsar#sending-messages-with-cached-producer), for example. However, if you aren't using Reactive Streams it would be an overkill to use the library only for producer caching. 
   
   Here is a code example in the Pulsar code base of producer caching:
   https://github.com/apache/pulsar/blob/18f2f4a9c1dab7eec5c7c9590b76aca17ee1ead8/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L148-L170
   
   The problem with that solution is that the producers accumulate and don't get closed automatically. Closing idle producers can be achieved by using [Caffeine cache](https://github.com/ben-manes/caffeine) as the "container". Caffeine's `removalListener` feature can be used to close the producer before disposing it.
   
   I hope this helps implementing your use case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org