You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/20 08:23:27 UTC
[GitHub] [pulsar] wuYin opened a new issue #7860: Message expiry may not effective
wuYin opened a new issue #7860:
URL: https://github.com/apache/pulsar/issues/7860
**Describe the bug**
unACK messages not be deleted after TTL
**To Reproduce**
1. Deploy a local cluster with default config: 3\*bookie, 1\*broker, 1\*topic, 1\*partition
2. Configure retention and TTL:
PS: Maybe I missed some configurations, correct me please, thanks
- retention: time 2min, size 10MB,
- TTL: 2min
3. Run client demos:
- fast produer, such as 2MB/s
- slow consumer, such as 80KB/s
```
{
"msgThroughputOut": 83840.243,
"msgThroughputIn": 2618852.783,
"bytesOutCount": 35204500,
"bytesInCount": 1321704278,
"storageSize": 1321704278,
"backlogSize": 465145578,
"msgRateIn": 260.297,
"msgRateOut": 8.333
}
```
4. Waiting for delete unACK messages after TTL
5. After 2min, expired messages all remained in bookie journal directories:
**Expected behavior**
delete expired unACK messages after TTL
**Desktop**
- OS: macOS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wuYin commented on issue #7860: Message expiry may not effective
Posted by GitBox <gi...@apache.org>.
wuYin commented on issue #7860:
URL: https://github.com/apache/pulsar/issues/7860#issuecomment-678017589
thanks for your reply, I'll reduce `gcWaitTime` and `minorCompactionInterval` and try again.
not enough to understand these configurations, that's my fault.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on issue #7860: Message expiry may not effective
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #7860:
URL: https://github.com/apache/pulsar/issues/7860#issuecomment-677993672
@wuYin The files in Bookie is cleaned by "compaction" settings. like
```
#############################################################################
## Entry log compaction settings
#############################################################################
# Set the rate at which compaction will readd entries. The unit is adds per second.
compactionRate=1000
# If bookie is using hostname for registration and in ledger metadata then
# whether to use short hostname or FQDN hostname. Defaults to false.
# useShortHostName=false
# Threshold of minor compaction
# For those entry log files whose remaining size percentage reaches below
# this threshold will be compacted in a minor compaction.
# If it is set to less than zero, the minor compaction is disabled.
minorCompactionThreshold=0.2
# Interval to run minor compaction, in seconds
# If it is set to less than zero, the minor compaction is disabled.
# Note: should be greater than gcWaitTime.
minorCompactionInterval=3600
# Set the maximum number of entries which can be compacted without flushing.
# When compacting, the entries are written to the entrylog and the new offsets
# are cached in memory. Once the entrylog is flushed the index is updated with
# the new offsets. This parameter controls the number of entries added to the
# entrylog before a flush is forced. A higher value for this parameter means
# more memory will be used for offsets. Each offset consists of 3 longs.
# This parameter should _not_ be modified unless you know what you're doing.
# The default is 100,000.
compactionMaxOutstandingRequests=100000
# Threshold of major compaction
# For those entry log files whose remaining size percentage reaches below
# this threshold will be compacted in a major compaction.
# Those entry log files whose remaining size percentage is still
# higher than the threshold will never be compacted.
# If it is set to less than zero, the minor compaction is disabled.
majorCompactionThreshold=0.5
# Interval to run major compaction, in seconds
# If it is set to less than zero, the major compaction is disabled.
# Note: should be greater than gcWaitTime.
majorCompactionInterval=86400
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wuYin closed issue #7860: Message expiry may not effective
Posted by GitBox <gi...@apache.org>.
wuYin closed issue #7860:
URL: https://github.com/apache/pulsar/issues/7860
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org