You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/07/21 09:12:29 UTC
[GitHub] [rocketmq-connect] zhangqiufeng88 opened a new issue, #210: 数据无法同步问题
zhangqiufeng88 opened a new issue, #210:
URL: https://github.com/apache/rocketmq-connect/issues/210
1. 场景是使用rocketmq-opeartor部署了两个rocketmq集群A和B， 将rocketmq-connect容器化部署到集群与rocketmq集群A,B相同的k8s集群中，需求是将rocketmq集群A中的topic同步到rocket集群B.
部署rocketmq-connect时connector-cluster-topic,connector-config-topic,connector-position-topic,connector-offset-topic,replicator-store-topic 这几个topic在集群B已创建
2. rocketmq-connect配置文件是：
# rockermq-connect-runtime-config
workerId=standalone-worker
storePathRootDir=/rocketmq-connect/storeRoot
## Http port for user to access REST API
httpPort=8082
# Rocketmq namesrvAddr
namesrvAddr={集群B的namesrvAddr}
# RocketMQ acl
aclEnable=false
accessKey=rocketmq
secretKey=12345678
#autoCreateGroupEnable=false
clusterName="rocketmqB"
# Source or sink connector jar file dir,The default value is rocketmq-connect-sample
pluginPaths=/rocketmq-connect/connectors/rocketmq-replicator/target/rocketmq-replicator-0.1.0-SNAPSHOT-jar-with-dependencies.jar
3.发送请求给rocketmq-runtime,请求内容：
http://{runtimeip}:{runtime-port}/connectors/replicator-test?config={"connector-class":"org.apache.rocketmq.replicator.RmqSourceReplicator","source-rocketmq":"{集群A的namesrvAddr}","target-rocketmq":"{集群B的namesrvAddr}","replicator-store-topic":"replicatorTopic","task-divide-strategy":"0","white-list":"test-2","task-parallelism":"2","source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter","target-cluster":"rocketmqB","source-cluster":"rocketmqA","connect-topicname":"","topic.rename.format": "rename-${topic}"}
4. 看日志，在集群A和B已成功注册生产者，runtime日志也显示已经连接A,B的broker成功，问题是，topic没有成功从A同步到B集群。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] sunxiaojian commented on issue #210: 数据无法同步问题
Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #210:
URL: https://github.com/apache/rocketmq-connect/issues/210#issuecomment-1194897686
> >
>
> @sunxiaojian converter的配置跟你发的是一样的，是有其他地方写错吗
org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter 改成这个，我写错了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] sunxiaojian closed issue #210: 数据无法同步问题
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian closed issue #210: 数据无法同步问题
URL: https://github.com/apache/rocketmq-connect/issues/210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] python-chun commented on issue #210: 数据无法同步问题
Posted by "python-chun (via GitHub)" <gi...@apache.org>.
python-chun commented on issue #210:
URL: https://github.com/apache/rocketmq-connect/issues/210#issuecomment-1553918084
使用RmqSourceReplicator，同步两个rocketmq集群(A,B)数据，只能将worker的nameSrv指定为B集群，并且task任务中source-rocketmq是A集群，target-rocketmq是B集群，这样部署吗？也就是只能source不能sink。当我把task任务中source-rocketmq改为B集群和target-rocketmq改为A集群，数据就不同步了；或者把worker的nameSrv指定为A集群，同样消息也不同步了。
请问如何实现sink的方式，因为我打算将A集群的消息同步给B集群和C集群，如果按照source的实现，我需要在B和C分别部署runtime worker才能实现。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] zhangqiufeng88 commented on issue #210: 数据无法同步问题
Posted by GitBox <gi...@apache.org>.
zhangqiufeng88 commented on issue #210:
URL: https://github.com/apache/rocketmq-connect/issues/210#issuecomment-1194889401
>
@sunxiaojian converter的配置跟你发的是一样的，是有其他地方写错吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-connect] sunxiaojian commented on issue #210: 数据无法同步问题
Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #210:
URL: https://github.com/apache/rocketmq-connect/issues/210#issuecomment-1193894990
> 1. 场景是使用rocketmq-opeartor部署了两个rocketmq集群A和B， 将rocketmq-connect容器化部署到集群与rocketmq集群A,B相同的k8s集群中，需求是将rocketmq集群A中的topic同步到rocket集群B.
> 部署rocketmq-connect时connector-cluster-topic,connector-config-topic,connector-position-topic,connector-offset-topic,replicator-store-topic 这几个topic在集群B已创建
> 2. rocketmq-connect配置文件是：
>
> # rockermq-connect-runtime-config
> workerId=standalone-worker storePathRootDir=/rocketmq-connect/storeRoot
>
> ## Http port for user to access REST API
> httpPort=8082
>
> # Rocketmq namesrvAddr
> namesrvAddr={集群B的namesrvAddr}
>
> # RocketMQ acl
> aclEnable=false accessKey=rocketmq secretKey=12345678
>
> #autoCreateGroupEnable=false clusterName="rocketmqB"
>
> # Source or sink connector jar file dir,The default value is rocketmq-connect-sample
> pluginPaths=/rocketmq-connect/connectors/rocketmq-replicator/target/rocketmq-replicator-0.1.0-SNAPSHOT-jar-with-dependencies.jar
>
> 3.发送请求给rocketmq-runtime,请求内容： http://{runtimeip}:{runtime-port}/connectors/replicator-test?config={"connector-class":"org.apache.rocketmq.replicator.RmqSourceReplicator","source-rocketmq":"{集群A的namesrvAddr}","target-rocketmq":"{集群B的namesrvAddr}","replicator-store-topic":"replicatorTopic","task-divide-strategy":"0","white-list":"test-2","task-parallelism":"2","source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter","target-cluster":"rocketmqB","source-cluster":"rocketmqA","connect-topicname":"","topic.rename.format": "rename-${topic}"} 4. 看日志，在集群A和B已成功注册生产者，runtime日志也显示已经连接A,B的broker成功，问题是，topic没有成功从A同步到B集群。
converter 配置有误，改成 "source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter",
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org