You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/09 14:39:15 UTC
[GitHub] [pulsar] beyondyinjl2 created a discussion: How to dynamically produce and consume topic data
GitHub user beyondyinjl2 created a discussion: How to dynamically produce and consume topic data
After creating a Producer, I want to send data to multiple different topics, or create a Consumer, I want to consume data from multiple topics
GitHub link: https://github.com/apache/pulsar/discussions/18853
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] lhotari added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: How to dynamically produce and consume topic data
> When producing data, I want to dynamically send it to different topics. The topics may be in different namespaces and different tenants.
In Pulsar client, you must create a separate producer for sending to a different topic. There are many ways to achieve what you are describing. The simplest approach is to create a producer, send the message and then close the producer. However, for most cases, this wouldn't be efficient, and producer caching is needed.
This challenge is handled in the [Reactive Streams adapter for Pulsar Java Client](https://github.com/lhotari/reactive-pulsar#sending-messages-with-cached-producer), for example. However, if you aren't using Reactive Streams it would be an overkill to use the library only for producer caching.
Here is a code example in the Pulsar code base of producer caching:
https://github.com/apache/pulsar/blob/18f2f4a9c1dab7eec5c7c9590b76aca17ee1ead8/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L148-L170
The problem with that solution is that the producers accumulate and don't get closed automatically. Closing idle producers can be achieved by using [Caffeine cache](https://github.com/ben-manes/caffeine) as the "container". Caffeine's `removalListener` feature can be used to close the producer before disposing it.
I hope this helps implementing your use case.
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352695
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
Like mqtt, a producer can send data on any topic, and a consumer can subscribe to topics dynamically
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352693
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] liangyuanpeng added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user liangyuanpeng added a comment to the discussion: How to dynamically produce and consume topic data
You can use `client.newConsumer().topicsPattern()` for multiple topics.
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352694
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
When producing data, I want to dynamically send it to different topics. The topics may be in different namespaces and different tenants.
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352692
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user github-actions[bot] added a comment to the discussion: How to dynamically produce and consume topic data
The issue had no activity for 30 days, mark with Stale label.
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352696
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
Posted by GitBox <gi...@apache.org>.
GitHub user beyondyinjl2 added a comment to the discussion: How to dynamically produce and consume topic data
For example, first subscribe to topic:persistent://public/default/my-topic:
Consumer consumer = client.newConsumer()
.topic("persistent://public/default/my-topic")
.subscriptionName("my-subscription")
.subscribe();
Then I want to dynamically subscribe to other topics: persistent://public/default/aaa or persistent://public/mynamesapce/bbb without restarting the service
GitHub link: https://github.com/apache/pulsar/discussions/18853#discussioncomment-4352691
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org