You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "sunxiaojian (via GitHub)" <gi...@apache.org> on 2023/03/01 10:00:43 UTC
[GitHub] [rocketmq] sunxiaojian opened a new pull request, #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
sunxiaojian opened a new pull request, #6219:
URL: https://github.com/apache/rocketmq/pull/6219
Changelog :
1. Modify the client. When the key is not empty, it will not be checked whether the body is empty; Support adding messages with empty body
2. Add the delete.deletion.ms attribute when creating a topic; Empty messages will not be cleared directly during the compression process. Only when the configured timeout is reached will they be cleared during the compression process; If the compression process is cleared directly, the consumer may not consume and the client cache cannot be updated
3.When incremental compact is performed, even if the incremental file is empty, the compact operation will continue. In order to clear the expired empty message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1463393653
> > > > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
> > >
> > >
> > > 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
> > > > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
> > >
> > >
> > > 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
> >
> >
> > 虽说是个kv系统，但不应该完全按照一个kv系统看待，实际的的kv系统是个共享存储，客户端直接请求远程获取的，所以正常的Kv系统是数据更新即可见；而compact topic 更多的场景是初始化加载全量，然后增量订阅变更（最大区别），客户端一般也不会直接去broker获取key相关的信息， 这个特性造成了，存储->订阅更新完成 有个过程， 直接删除null消息就会出现客户端订阅不到变更的问题，也就无法感知删除了； 其次，【在本地先compaction一次】，只适用于单client的情况下，应该对于分布式环境不适用【若在应用中再附以共享存储，那就没必要使用compaction topic了】 对于topic/partition比较多时，可能占用资源较高，这个还需要测试来分析，但我认为，compaction频率不需要太频繁，目前的默认值还是有点频繁的(可选择集群低峰期压缩)； 也可以将 开启删
除 作为一个可配置的功能加入，若不开启，则保持原逻辑不变; 当然也可以高峰期依旧保持增量，低峰期定时全量一次（能保证一天一次全量即可），来解决性能的问题
>
> 想了下，确实是存在null消息在消费到之前就被删除的情况，确实是需要增加一定的保留时间，并且看起来null消息不同的保留时长并不会带来什么影响，那是不是说可以在compact时判断下是否需要保留null消息即可呢
是的，但我也加了在topic不活跃的情况下，块文件新增可能比较慢，这样的情况下我加了空跑次数来合并清除历史的空数据，看看这部分需不需要，不加应该也可以接受
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1463331230
>
是的，但我也加了在topic不活跃的情况下，合并区块文件新增可能比较慢，这样的情况下我加了空跑次数来合并清除历史的空数据，看看这部分需不需要，不加应该也可以接受
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] fuyou001 commented on a diff in pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "fuyou001 (via GitHub)" <gi...@apache.org>.
fuyou001 commented on code in PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#discussion_r1123962885
##########
store/src/main/java/org/apache/rocketmq/store/kv/CompactionLog.java:
##########
@@ -327,13 +336,20 @@ boolean shouldRetainMsg(final MessageExt msgExt, final OffsetMap map) throws Dig
if (StringUtils.isNotBlank(key)) {
boolean keyNotExistOrOffsetBigger = msgExt.getQueueOffset() >= map.get(key);
boolean hasBody = ArrayUtils.isNotEmpty(msgExt.getBody());
- return keyNotExistOrOffsetBigger && hasBody;
+ return keyNotExistOrOffsetBigger && !verifyDeleted(hasBody, msgExt);
} else {
log.error("message has no keys");
Review Comment:
log messageId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #5779]RocketMQ compaction topic support tombstone messages [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-2038478551
This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #5779]RocketMQ compaction topic support tombstone messages [rocketmq]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian closed pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
URL: https://github.com/apache/rocketmq/pull/6219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1455375338
> > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
>
> 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
>
> > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
>
> 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
> > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
>
> 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
>
> > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
>
> 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
关于2:
> > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
>
> 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
>
> > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
>
> 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
【上面是我对compaction topic的想法】：
虽说是个kv系统，但不应该完全按照一个kv系统看待，实际的的kv系统是个共享存储，直接远程获取的，所以正常的Kv系统是数据更新即可见；而compact topic 更多的场景是初始化加载全量，然后增量订阅变更（最大区别），客户端一般也不会直接去broker获取key相关的信息， 这个特性造成了，存储->订阅更新完成 有个过程， 直接删除null消息就会出现客户端订阅不到变更的问题，也就无法感知删除了；
其次，【在本地先compaction一次】，只适用于单client的情况下，应该对于分布式环境不适用【若在应用中再附以共享存储，那就没必要使用compaction topic了】
对于topic/partition比较多时，可能占用资源较高，这个还需要测试来分析，但我个人任务，compaction频率不需要太频繁，目前的默认还是有点频繁的；
其次，可以将开启删除作为一个可配置的功能加入，若不开启，则保持原逻辑不变
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1451193869
# [Codecov](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6219](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (baf4179) into [develop](https://codecov.io/gh/apache/rocketmq/commit/bf1131113596582b76d0e3172c2de544a54cf2c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf11311) will **decrease** coverage by `0.06%`.
> The diff coverage is `37.50%`.
> :exclamation: Current head baf4179 differs from pull request most recent head 0fc41f8. Consider uploading reports for the commit 0fc41f8 to get more accurate results
```diff
@@ Coverage Diff @@
## develop #6219 +/- ##
=============================================
- Coverage 43.24% 43.19% -0.06%
+ Complexity 8859 8856 -3
=============================================
Files 1094 1094
Lines 77148 77191 +43
Branches 10062 10074 +12
=============================================
- Hits 33366 33343 -23
- Misses 39612 39678 +66
Partials 4170 4170
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...broker/processor/AbstractSendMessageProcessor.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0Fic3RyYWN0U2VuZE1lc3NhZ2VQcm9jZXNzb3IuamF2YQ==) | `32.91% <0.00%> (-0.32%)` | :arrow_down: |
| [...va/org/apache/rocketmq/store/kv/CompactionLog.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2t2L0NvbXBhY3Rpb25Mb2cuamF2YQ==) | `21.09% <0.00%> (-0.22%)` | :arrow_down: |
| [...ls/command/message/PrintMessageByQueueCommand.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Rvb2xzL2NvbW1hbmQvbWVzc2FnZS9QcmludE1lc3NhZ2VCeVF1ZXVlQ29tbWFuZC5qYXZh) | `3.10% <0.00%> (-0.78%)` | :arrow_down: |
| [...in/java/org/apache/rocketmq/client/Validators.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvVmFsaWRhdG9ycy5qYXZh) | `67.39% <62.50%> (+0.72%)` | :arrow_up: |
| [...ava/org/apache/rocketmq/broker/util/HookUtils.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvdXRpbC9Ib29rVXRpbHMuamF2YQ==) | `21.23% <66.66%> (-1.37%)` | :arrow_down: |
| [...va/org/apache/rocketmq/common/TopicAttributes.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vVG9waWNBdHRyaWJ1dGVzLmphdmE=) | `93.33% <100.00%> (+1.02%)` | :arrow_up: |
| [...q/namesrv/routeinfo/BrokerHousekeepingService.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bmFtZXNydi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvbmFtZXNydi9yb3V0ZWluZm8vQnJva2VySG91c2VrZWVwaW5nU2VydmljZS5qYXZh) | `70.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...org/apache/rocketmq/store/ha/WaitNotifyObject.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL1dhaXROb3RpZnlPYmplY3QuamF2YQ==) | `80.35% <0.00%> (-5.36%)` | :arrow_down: |
| [...controller/impl/DefaultBrokerHeartbeatManager.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvY29udHJvbGxlci9pbXBsL0RlZmF1bHRCcm9rZXJIZWFydGJlYXRNYW5hZ2VyLmphdmE=) | `82.27% <0.00%> (-3.80%)` | :arrow_down: |
| [...mq/broker/topic/TopicQueueMappingCleanService.java](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvdG9waWMvVG9waWNRdWV1ZU1hcHBpbmdDbGVhblNlcnZpY2UuamF2YQ==) | `12.43% <0.00%> (-3.63%)` | :arrow_down: |
| ... and [44 more](https://codecov.io/gh/apache/rocketmq/pull/6219?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] fuyou001 commented on a diff in pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "fuyou001 (via GitHub)" <gi...@apache.org>.
fuyou001 commented on code in PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#discussion_r1123962408
##########
store/src/main/java/org/apache/rocketmq/store/kv/CompactionLog.java:
##########
@@ -327,13 +336,20 @@ boolean shouldRetainMsg(final MessageExt msgExt, final OffsetMap map) throws Dig
if (StringUtils.isNotBlank(key)) {
boolean keyNotExistOrOffsetBigger = msgExt.getQueueOffset() >= map.get(key);
boolean hasBody = ArrayUtils.isNotEmpty(msgExt.getBody());
- return keyNotExistOrOffsetBigger && hasBody;
+ return keyNotExistOrOffsetBigger && !verifyDeleted(hasBody, msgExt);
} else {
log.error("message has no keys");
Review Comment:
add messageId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1461603427
增加配置参数 `compactionScheduleIdleTimes 表示当没有新增数据时的调度空跑次数`，
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ltamber commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "ltamber (via GitHub)" <gi...@apache.org>.
ltamber commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1463143641
> > > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
> >
> >
> > 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
> > > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
> >
> >
> > 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
>
> 虽说是个kv系统，但不应该完全按照一个kv系统看待，实际的的kv系统是个共享存储，客户端直接请求远程获取的，所以正常的Kv系统是数据更新即可见；而compact topic 更多的场景是初始化加载全量，然后增量订阅变更（最大区别），客户端一般也不会直接去broker获取key相关的信息， 这个特性造成了，存储->订阅更新完成 有个过程， 直接删除null消息就会出现客户端订阅不到变更的问题，也就无法感知删除了； 其次，【在本地先compaction一次】，只适用于单client的情况下，应该对于分布式环境不适用【若在应用中再附以共享存储，那就没必要使用compaction topic了】 对于topic/partition比较多时，可能占用资源较高，这个还需要测试来分析，但我认为，compaction频率不需要太频繁，目前的默认值还是有点频繁的(可选择集群低峰期压缩)； 也可以将 开启删�
� 作为一个可配置的功能加入，若不开启，则保持原逻辑不变; 当然也可以高峰期依旧保持增量，低峰期定时全量一次（能保证一天一次全量即可），来解决性能的问题
想了下，确实是存在null消息在消费到之前就被删除的情况，确实是需要增加一定的保留时间，并且看起来null消息不同的保留时长并不会带来什么影响，那是不是说可以在compact时判断下是否需要保留null消息即可呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1449772482
#5779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ltamber commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "ltamber (via GitHub)" <gi...@apache.org>.
ltamber commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1452890557
> 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间；
另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
> 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1455376906
> > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
>
> 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
>
> > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
>
> 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
虽说是个kv系统，但不应该完全按照一个kv系统看待，实际的的kv系统是个共享存储，直接远程获取的，所以正常的Kv系统是数据更新即可见；而compact topic 更多的场景是初始化加载全量，然后增量订阅变更（最大区别），客户端一般也不会直接去broker获取key相关的信息， 这个特性造成了，存储->订阅更新完成 有个过程， 直接删除null消息就会出现客户端订阅不到变更的问题，也就无法感知删除了；
其次，【在本地先compaction一次】，只适用于单client的情况下，应该对于分布式环境不适用【若在应用中再附以共享存储，那就没必要使用compaction topic了】
对于topic/partition比较多时，可能占用资源较高，这个还需要测试来分析，但我个人任务，compaction频率不需要太频繁，目前的默认还是有点频繁的；
其次，可以将开启删除作为一个可配置的功能加入，若不开启，则保持原逻辑不变
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1455378609
> > 2.创建topic时增加delete.deletion.ms属性，设置空消息的清除时间，不在合并中直接清除的直接原因，主要是担心消息发送和压缩过程太近，造成客户端还未消费到就被清除，实际应用中可能造成客户端无法更新本地数据
>
> 个人感觉这个属性`delete.retention.ms`名字可能会导致混淆，被认为是正常消息过期删除的时间； 另外对于一个类kv系统，我觉得delete之后就应该get不到，我们应该缩短这个耗时，而不是增加这个耗时，compaction topic的正确使用应该是消费到消息之后，在本地先compaction一次，也就是说null消息在客户端本地就应该把之前相同key的消息删除(覆盖)，从而达到一个**实时**效果，如果需要消费到中间数据，顺序消息是不是能更好的满足需求。
>
> > 3.增量compaction时，没有新增文件也会继续进行compact，主要是为了清除body为 null的消息
>
> 个人感觉这个过程，代价有点高，当compaction类型的topic/partition比较多时，这块占用资源会很高，当然如果基于上面说的，是否不需要这个过程
虽说是个kv系统，但不应该完全按照一个kv系统看待，实际的的kv系统是个共享存储，直接远程获取的，所以正常的Kv系统是数据更新即可见；而compact topic 更多的场景是初始化加载全量，然后增量订阅变更（最大区别），客户端一般也不会直接去broker获取key相关的信息， 这个特性造成了，存储->订阅更新完成 有个过程， 直接删除null消息就会出现客户端订阅不到变更的问题，也就无法感知删除了；
其次，【在本地先compaction一次】，只适用于单client的情况下，应该对于分布式环境不适用【若在应用中再附以共享存储，那就没必要使用compaction topic了】
对于topic/partition比较多时，可能占用资源较高，这个还需要测试来分析，但我个人任务，compaction频率不需要太频繁，目前的默认值还是有点频繁的； 也可以将 开启删除 作为一个可配置的功能加入，若不开启，则保持原逻辑不变
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] sunxiaojian commented on pull request #6219: [ISSUE #5779]RocketMQ compaction topic support tombstone messages
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #6219:
URL: https://github.com/apache/rocketmq/pull/6219#issuecomment-1496885280
@ltamber @fuyou001 辛苦继续帮忙review 这个 PR是否合理，还需要什么改动的地方，如果合理是否考虑合并或继续增加讨论
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org