You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/12/20 02:31:55 UTC
[GitHub] [rocketmq] iamqq23ue opened a new issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
iamqq23ue opened a new issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661
version:4.9.1
master: SYNC_MASTER slave:slave
Test steps: stop the slave, send a batch of data, start the slave, and find that the master and slave can synchronize, but only the latest files.
As a result, if the slave goes down for a period of time for some reason, the historical data cannot be synchronized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997776748
org.apache.rocketmq.store.ha.HAService.HAClient#reportSlaveMaxOffset
可以同步的。开始同步的位点都是slave自己上报的，master不会更改这个位点的。
你是怎么判断历史数据没有同步的呀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-1002409804
It is not recommended to directly change the slave, a large number of cold reads is likely to cause the mast broker to be unavailable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MatrixHB edited a comment on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
MatrixHB edited a comment on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997875813
look at org.apache.rocketmq.store.ha.HAService.HAClient#dispatchReadRequest
```
// 读取slave本地的最大偏移量
long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
if (slavePhyOffset != 0) {
// slave最大偏移量应该与master传过来的偏移量相等，保证同步消息连续
if (slavePhyOffset != masterPhyOffset) {
log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
+ slavePhyOffset + " MASTER: " + masterPhyOffset);
return false;
}
}
```
this ensures the slave offset should be continuous. Thesituation you describe is unlikely to happen, can you give more details ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 edited a comment on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
panzhi33 edited a comment on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997776748
org.apache.rocketmq.store.ha.HAService.HAClient#reportSlaveMaxOffset
可以同步的。开始同步的位点都是slave自己上报的，master不会更改这个位点的。
启动slave后，之前的数据还存在嘛？你是怎么判断历史数据没有同步的呀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-998514731
现象很明显，同步速度很快，而且slave上面目录大小600M（master 100G）,明显是没有同步
------------------ 原始邮件 ------------------
发件人: "apache/rocketmq" ***@***.***>;
发送时间: 2021年12月20日(星期一) 晚上6:05
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661)
org.apache.rocketmq.store.ha.HAService.HAClient#reportSlaveMaxOffset
可以同步的。开始同步的位点都是slave自己上报的，master不会更改这个位点的。
你是怎么判断历史数据没有同步的呀
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MatrixHB commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
MatrixHB commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997875813
look at org.apache.rocketmq.store.ha.HAService.HAClient#dispatchReadRequest
```
long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
if (slavePhyOffset != 0) {
if (slavePhyOffset != masterPhyOffset) {
log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
+ slavePhyOffset + " MASTER: " + masterPhyOffset);
return false;
}
}
```
this ensures the slave offset should be continuous. Thesituation you describe is unlikely to happen, can you give more details ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MatrixHB edited a comment on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
MatrixHB edited a comment on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997875813
look at org.apache.rocketmq.store.ha.HAService.HAClient#dispatchReadRequest
```
// 读取slave本地的最大偏移量
long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
if (slavePhyOffset != 0) {
// slave最大偏移量应该与master传过来的偏移量相等，保证同步消息连续
if (slavePhyOffset != masterPhyOffset) {
log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
+ slavePhyOffset + " MASTER: " + masterPhyOffset);
return false;
}
}
```
this ensures the slave offset should be continuous. The situation you describe is unlikely to happen, can you give more details ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-1001894165
I understand that this problem occurs in two situations.
- 1. The stand-alone machine changes to m-s mode. This is usually only a test, and it is generally unlikely to exist online.
- 2. There are multiple groups of active and standby brokers online (at least 2m2s is recommended), and the standby machine is faulty and the standby machine needs to be replaced. If this happens, it is recommended that the master broker set write prohibition, and after 3 days (determine the specific time according to the setting message save time), rebuild the group of brokers.
If you change the slave, start synchronization from the broker, because the amount of synchronization is large, it is likely that the master broker is busy in synchronization, which makes the broker unavailable, so it is not recommended to replace the slave in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-1001930649
I understand that this problem occurs in two situations.
The stand-alone machine changes to m-s mode. This is usually only a test, and it is generally unlikely to exist online.
There are multiple groups of active and standby brokers online (at least 2m2s is recommended), and the standby machine is faulty and the standby machine needs to be replaced. If this happens, it is recommended that the master broker set write prohibition, and after 3 days (determine the specific time according to the setting message save time), rebuild the group of brokers.
If you change the slave, start synchronization from the broker, because the amount of synchronization is large, it is likely that the master broker is busy in synchronization, which makes the broker unavailable, so it is not recommended to replace the slave in this way.
您好。在我看来，如果一个slave故障且无法恢复，我们可以直接创建一个新的slave同步数据，否则单点运行几天更不适合实际的生产环境。
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997723075
确实是只同步最新的一个文件。但是为何slave不能同步历史的数据呢？master上的数据都在，没有被删除。
------------------ 原始邮件 ------------------
发件人: "apache/rocketmq" ***@***.***>;
发送时间: 2021年12月20日(星期一) 中午11:35
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661)
If the slave's maxPhyOffset can find the message data in the master, it will start to synchronize the message from this place. If it can't be found, it will not be synchronized. It’s not like what you said, it’s not synchronized in the middle, only the latest
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-998554141
这个设计有个问题，我如果想要增加一个slave，或者现有的slave坏了，就没办法同步历史数据了
--------------------------------------
这个是预期内的，如果之前没有同步过，那么就会从master的最新也就是最大位点开始。
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-999331591
>
> 这个设计有个问题，我如果想要增加一个slave，或者现有的slave坏了，就没办法同步历史数据了
> […](#)
> -------------------------------------- 这个是预期内的，如果之前没有同步过，那么就会从master的最新也就是最大位点开始。 — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: ***@***.***>
是的，我后面提个pr进行增强，这里可以加个配置，可以选择从最小或者最大位点进行同步。然后文档进行说明，如果从最小位点开始同步，需要对master禁写，否则同步期间master会产生大量冷数据，对消息的收发都有影响的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-1002411301
For cold data synchronization, we can set the master to read-only and restore it to writable after synchronization.
In addition, I also found that if there are two slaves, and one of them lags too far behind, only the latest file will be synchronized.
------------------ 原始邮件 ------------------
发件人: "apache/rocketmq" ***@***.***>;
发送时间: 2021年12月29日(星期三) 中午1:57
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661)
It is not recommended to directly change the slave, a large number of cold reads is likely to cause the mast broker to be unavailable
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-997565581
If the slave's maxPhyOffset can find the message data in the master, it will start to synchronize the message from this place. If it can't be found, it will not be synchronized. It’s not like what you said, it’s not synchronized in the middle, only the latest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-999334047
谢谢，期待您的增强
------------------ 原始邮件 ------------------
发件人: "apache/rocketmq" ***@***.***>;
发送时间: 2021年12月22日(星期三) 下午2:56
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661)
这个设计有个问题，我如果想要增加一个slave，或者现有的slave坏了，就没办法同步历史数据了
…
-------------------------------------- 这个是预期内的，如果之前没有同步过，那么就会从master的最新也就是最大位点开始。 — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: @.***>
是的，我后面提个pr进行增强，这里可以加个配置，可以选择从最小或者最大位点进行同步。然后文档进行说明，如果从最小位点开始同步，需要对master禁写，否则同步期间master会产生大量冷数据，对消息的收发都有影响的。
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-998534528
> 可以按如下方法复现只同步最新commitlog的情况 1，先不启动slave，在一个全新的master上创建topic生产一段时间，让主从差距有个上百G，然后启动slave。
> […](#)
> ------------------ 原始邮件 ------------------ 发件人: "apache/rocketmq" ***@***.***>; 发送时间: 2021年12月20日(星期一) 晚上8:21 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661) look at org.apache.rocketmq.store.ha.HAService.HAClient#dispatchReadRequest long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset(); if (slavePhyOffset != 0) { if (slavePhyOffset != masterPhyOffset) { log.error("master pushed offset not equal the max phy offset in slave, SLAVE: " + slavePhyOffset + " MASTER: " + masterPhyOffset); return false; } } this ensures the slave offset should be continuous. Thesituation you describe is unlikely to happen, can you give more details ?? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
这个是预期内的，如果之前没有同步过，那么就会从master的最新也就是最大位点开始。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamqq23ue commented on issue #3661: Why only the last few files are synchronized when the master-slave synchronization gap is large
Posted by GitBox <gi...@apache.org>.
iamqq23ue commented on issue #3661:
URL: https://github.com/apache/rocketmq/issues/3661#issuecomment-998516157
可以按如下方法复现只同步最新commitlog的情况
1，先不启动slave，在一个全新的master上创建topic生产一段时间，让主从差距有个上百G，然后启动slave。
------------------ 原始邮件 ------------------
发件人: "apache/rocketmq" ***@***.***>;
发送时间: 2021年12月20日(星期一) 晚上8:21
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/rocketmq] Why only the last few files are synchronized when the master-slave synchronization gap is large (Issue #3661)
look at org.apache.rocketmq.store.ha.HAService.HAClient#dispatchReadRequest
long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset(); if (slavePhyOffset != 0) { if (slavePhyOffset != masterPhyOffset) { log.error("master pushed offset not equal the max phy offset in slave, SLAVE: " + slavePhyOffset + " MASTER: " + masterPhyOffset); return false; } }
this ensures the slave offset should be continuous. Thesituation you describe is unlikely to happen, can you give more details ??
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org