[GitHub] [pulsar] zbentley commented on issue #15399: Does plusar support key-shared producer?

[GitBox <gi...@apache.org>]

zbentley commented on issue #15399:
URL: https://github.com/apache/pulsar/issues/15399#issuecomment-1114208054

   That makes sense. The second answer I posted above is applicable, then. You should either:
   - Add partitions to the topic.
   - Use a key-based batcher in your producer with multiple IO threads.
   - Use multiple producers.
   
   Unless you use multiple partitions (or multiple topics), Pulsar will still totally order your messages according to their timestamps. If you truly want these messages to be handled fully in parallel at every level (i.e. they can be received by Pulsar in parallel, stored to BookKeeper in parallel, etc.), you must use multiple partitions or multiple topics.
   
   With multiple partitions and a key-based batcher, there is a high probability of the full parallelism you describe wanting. With multiple topics and multiple producers (or multiple producers on a partitioned topic with a custom batcher which guarnatees that `key1` and `key2` go to separate partitions) that probability approaches 100%. However, in my opinion the latter approach increases complexity for minimal gain.
   
   
   However, I would urge you to benchmark to be sure that there really is a limitation on throughput in your base case. Pulsar's internal imposition of global ordering and store operations are already highly performant and parallelized internally. If you are seeing bottlenecks, I suspect that it is not due to "global ordering" being imposed on messages, and is instead due to the throughput limitations of a single-partition topic in general (likely at the storage layer; try running a reduced persistence config or a non-persistent topic to prove this out), in which case partitioning is the way to go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org