You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/29 08:34:21 UTC
[GitHub] [pulsar] trexinc opened a new issue #7095: Support key shared batching in Functions
trexinc opened a new issue #7095:
URL: https://github.com/apache/pulsar/issues/7095
There is a publisher that publishes with keys (and key based batching enabled) to a certain partitioned topic.
A routing function (see below) reads from this topic and forwards the messages to a different partitioned topic (keeping the key with slight modification). If I now consume with KEY_SHARED from this topic then only one consumer will get all the messages.
```python
from pulsar import Function
class RoutingFunction(Function):
def process(self, item, context):
properties = context.get_message_properties()
context.publish("persistent://" + properties["tenant"] + "/ns/" + properties["dst_id"], item, message_conf={"partition_key": properties["tenant"] + "_" + context.get_partition_key()})
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] trexinc commented on issue #7095: Support key shared batching in Functions
Posted by GitBox <gi...@apache.org>.
trexinc commented on issue #7095:
URL: https://github.com/apache/pulsar/issues/7095#issuecomment-635883555
Seems that this feature is not overly complicated to implement, just add a config to function to enable key based batching, and then create producers according to this config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zongtanghu commented on issue #7095: Support key shared batching in Functions
Posted by GitBox <gi...@apache.org>.
zongtanghu commented on issue #7095:
URL: https://github.com/apache/pulsar/issues/7095#issuecomment-647140054
@codelipenghui @zhaijack @sijie Has anyone else to resolve this feature?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] trexinc commented on issue #7095: Support key shared batching in Functions
Posted by GitBox <gi...@apache.org>.
trexinc commented on issue #7095:
URL: https://github.com/apache/pulsar/issues/7095#issuecomment-636287782
In addition, KEY_SHARED mode should be added to function consuming as well, as it is most useful for improving performance with --retain-ordering & --parallelism
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org