You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by "efcasado (via GitHub)" <gi...@apache.org> on 2026/10/17 10:54:33 UTC

[PR] [improve][io] Support regex message keys in RabbitMQ source [pulsar-connectors]

efcasado opened a new pull request, #140:
URL: https://github.com/apache/pulsar-connectors/pull/140

### Motivation

Extend RabbitMQ source connector with support for deriving message keys from RabbitMQ routing keys using regex replacements.

This gives us more control over how RabbitMQ routing keys are mapped to Pulsar message keys.

We have a use case where RabbitMQ messages are published with routing keys of the form `<context>.<aggregate_id>`. Using `<context>.<aggregate_id>` as the message key does not guarantee that messages will be consumed in order in Pulsar. Even in a single-partition topic, `foo.123` and `bar.123` could be consumed out of order by different key-shared consumers.

The idea is to let users define an optional list of rules to derive message keys from routing keys. See below for an example

```yaml
messageKeyRules:
- regex: '(foo|bar)\.([0-9]+)'
replacement: '$1'
```

The original routing key is preserved as a message property, i.e. `__rabbitmq_routing_key`. Same convention we used in https://github.com/apache/pulsar-connectors/pull/90 to preserve other metadata such as consumer tag and queue name.

If no rules are provided, the old behaviour is preserved - i.e. using the RabbitMQ routing key as message key. Unmatched routing keys preserve the old behaviour too. Thus, this is a non-breaking and backwards compatible change.

### Modifications

`RabbitMQSourceConfig.java` has been extended with an optional `messageKeyRules` configuration parameter, which accepts regex-replacement pairs.

`RabbitMQSource.java` has been extended to compile the regex rules at startup, and to derive the message key from the routing key when there is a matching rule. It also preserves the original RabbitMQ routign key as a `__rabbitmq_routing_key` message property.

### Verifying this change

- [x] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

- Extended the existing `RabbitMQSourceConfigTest` and `RabbitMQSourceTest` tests with additional tests to confirm the new configuration parameters are correctly applied and the AMQP fields are correctly propagated to the resulting Pulsar message.

### Documentation

- [ ] `doc` 
- [ ] `doc-required` 
- [x] `doc-not-needed` 
- [ ] `doc-complete`

### Matching PR in forked repository

PR in forked repository: https://github.com/efcasado/pulsar-connectors/pull/4

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][rmq] Support regex message keys [pulsar-connectors]

Posted by "efcasado (via GitHub)" <gi...@apache.org>.

efcasado commented on PR #140:
URL: https://github.com/apache/pulsar-connectors/pull/140#issuecomment-6271799694

   You were faster than me, @david-streamlio! 😛
   
   I wanted to thank you for the thorough and quick feedback. I ended up settling for using empty keys because it feels more intentional than falling back to the original routing key. If someone wants to fall back to the original routing key, they can tighten the regex and then the message goes unmatched, etc.
   
   Glad to see this change getting merged. It's certainly very useful for us, and we hope it helps others too! 🙏 
   
   [Here](https://github.com/apache/pulsar-site/pull/1257) is the pull request to update the docs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][rmq] Support regex message keys [pulsar-connectors]

Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.

david-streamlio merged PR #140:
URL: https://github.com/apache/pulsar-connectors/pull/140


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][io] Support regex message keys in RabbitMQ source [pulsar-connectors]

Posted by "efcasado (via GitHub)" <gi...@apache.org>.

efcasado commented on PR #140:
URL: https://github.com/apache/pulsar-connectors/pull/140#issuecomment-6233160846

   @david-streamlio - curious to hear your thoughts on this one.
   
   It solves a real problem we are facing when connecting our Pulsar cluster to an external RabbitMQ source. The "problem" is that the RabbitMQ source uses complex routing keys (see [here](https://docs.sportradar.com/uof/data-and-features/messages/amqp-topic-filtering) for an example) that cannot be used as-is as Pulsar message keys. Or, they can be used, but we would give up ordering guarantees when using a key-shared subscription, which is not ideal. The routing keys include an aggregate id that is the perfect value to use as message id for our use case.
   
   We believe the regex-based replacements is generic enough and others in the community may find it useful too. Let's see what you think about this 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org