You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/25 02:52:01 UTC
[GitHub] [rocketmq-connect] oudb opened a new issue, #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
oudb opened a new issue, #286:
URL: https://github.com/apache/rocketmq-connect/issues/286
The issue tracker is used for bug reporting purposes **ONLY** whereas feature request needs to follow the [RIP process](https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal). To avoid unnecessary duplication, please check whether there is a previous issue before filing a new one.
It is recommended to start a discussion thread in the [mailing lists](http://rocketmq.apache.org/about/contact/) in cases of discussing your deployment plan, API clarification, and other non-bug-reporting issues.
We welcome any friendly suggestions, bug fixes, collaboration, and other improvements.
Please ensure that your bug report is clear and self-contained. Otherwise, it would take additional rounds of communication, thus more time, to understand the problem itself.
Generally, fixing an issue goes through the following steps:
1. Understand the issue reported;
1. Reproduce the unexpected behavior locally;
1. Perform root cause analysis to identify the underlying problem;
1. Create test cases to cover the identified problem;
1. Work out a solution to rectify the behavior and make the newly created test cases pass;
1. Make a pull request and go through peer review;
As a result, it would be very helpful yet challenging if you could provide an isolated project reproducing your reported issue. Anyway, please ensure your issue report is informative enough for the community to pick up. At a minimum, include the following hints:
**BUG REPORT**
1. Please describe the issue you observed:
- What did you do (The steps to reproduce)?
- What is expected to see?
- What did you see instead?
2. Please tell us about your environment:
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
**FEATURE REQUEST**
1. Please describe the feature you are requesting.
2. Provide any additional detail on your proposed use case for this feature.
2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
4. If there are some sub-tasks involved, use -[] for each sub-task and create a corresponding issue to map to the sub-task:
- [sub-task1-issue-number](example_sub_issue1_link_here): sub-task1 description here,
- [sub-task2-issue-number](example_sub_issue2_link_here): sub-task2 description here,
- ...
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] oudb commented on issue #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
Posted by GitBox <gi...@apache.org>.
oudb commented on issue #286:
URL: https://github.com/apache/rocketmq-connect/issues/286#issuecomment-1226725184
新增接口RocketmqRecordPartitionKafkaTopicPartitionMapper支持上述方案，详见分支：https://github.com/oudb/rocketmq-connect/tree/rocketmq-connect-kafka-connector-adapter-mongodb。 分支将新增kafka mongodb connector和kafka neo4j connector的文档。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] oudb commented on issue #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
Posted by GitBox <gi...@apache.org>.
oudb commented on issue #286:
URL: https://github.com/apache/rocketmq-connect/issues/286#issuecomment-1226716124
MessageQueue的brokerName可以编码到TopicPartition的topic或者partition。
方案一：当前版本编码到TopicPartition的topic：
（1）Kafka sink connector看到的是topic是topic和brokerName组合，这个组合是不可预测的，因为Rocketmq的brokerName是可变的，但是有些sink connector的配置项目需要提供具体的topic，比如neo4j connector： "neo4j.topic.cypher.my-topic": "MERGE (p:Person{name: event.name, surname: event.surname}) MERGE (f:Family{name: event.surname}) MERGE (p)-[:BELONGS_TO]->(f)"。
方案二：编码到TopicPartition的partition
给brokerName一个int的编号，然后编码到TopicPartition的partition前16bit里，基于大多数partition编号和broker数量都不会超过2^16=65536.
如何给brokerName一个int的编号
（1）提供一个配置项，用户指定：brokerName0:0,brokerName1:0
不足之处是当Rocketmq集群增加新节点的时候，需要重新提供新配置
（2）提供一个配置项，用户指定一个正则表达式，用户从brokerName提取编号，适用于brokerName带有编号的Rocketmq集群
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] oudb closed issue #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
Posted by GitBox <gi...@apache.org>.
oudb closed issue #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
URL: https://github.com/apache/rocketmq-connect/issues/286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] oudb commented on issue #286: kafka connector adapter：how map Rocketmq MessageQueue to Kafka TopicPartition
Posted by GitBox <gi...@apache.org>.
oudb commented on issue #286:
URL: https://github.com/apache/rocketmq-connect/issues/286#issuecomment-1226708939
the concept of Partition in Rocketmq is MessageQueue(topic,brokerName,queueId)，but in Kafka is TopicPartition(topic，partition)，we need some method to encode brokerName to TopicPartition：topic or partition. this is import for adapting Kafka sink connector, because sink connector will call the org.apache.kafka.connect.sink.SinkTaskContext,then will lost brokerName.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org