You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/01/22 04:36:15 UTC
[GitHub] [pulsar] lichuan6 opened a new issue #9271: topic retention not work
lichuan6 opened a new issue #9271:
URL: https://github.com/apache/pulsar/issues/9271
**Describe the bug**
I have a topic with `storageSize` equals to`194020994`
`pulsar-admin topics stats public/default/pulsar`
```json
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 194020994,
"msgInCounter" : 2460600,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 194020994,
"backlogSize" : 8473994,
"publishers" : [ ],
"subscriptions" : { },
"replication" : { },
"deduplicationStatus" : "Enabled"
}
```
then I set the retention with size 10M and time to zero.
`pulsar-admin namespaces set-retention --size 40M -t 0 public/default`
I expect to storage bytes will drop after setting the retention, but nothing changes after `topics stats` command call.
```json
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 194020994,
"msgInCounter" : 2460600,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 194020994,
"backlogSize" : 8473994,
"publishers" : [ ],
"subscriptions" : { },
"replication" : { },
"deduplicationStatus" : "Enabled"
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-767744226
@sijie please correct me if I'm wrong
@lichuan6 ManagedLedger consists of many BookKeeper ledgers, there'll be a current ledger that we write to. If curent ledger is closed(we shut down the service) or full a new ledger will be created and become the current ledger. There's a caveat that ledger trimming is performed on ledger level, e.g. it removes a whole ledger, and it **won't delete current ledger.** So in your case even if ledger retention is set to 10M and ledger size has exceeded, but it's the current ledger so it won't be removed. I think if you set `managedLedgerMaxSizePerLedgerMbytes` to lower value and produce messages to make topic write to a new ledger, you can observe the old ledger being deleted due to exceeded retention size limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lichuan6 commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
lichuan6 commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-771330258
@codelipenghui
Sorry for the late reply.
I notice there's a `managedLedgerMaxLedgerRolloverTimeMinutes` config which has a default value of 240(4hrs), which will close the managed ledger and trigger(I guess) the retention to work. The messages were deleted the next morning, apparently more than 4hrs.
@MarvinCai Thanks for your clarification, that helps a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lichuan6 commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
lichuan6 commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-771330258
@codelipenghui
Sorry for the late reply.
I notice there's a `managedLedgerMaxLedgerRolloverTimeMinutes` config which has a default value of 240(4hrs), which will close the managed ledger and trigger(I guess) the retention to work. The messages were deleted the next morning, apparently more than 4hrs.
@MarvinCai Thanks for your clarification, that helps a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-770229113
@lichuan6 Does the new update works for you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #9271:
URL: https://github.com/apache/pulsar/issues/9271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #9271:
URL: https://github.com/apache/pulsar/issues/9271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-767744226
@sijie please correct me if I'm wrong
@lichuan6 ManagedLedger consists of many BookKeeper ledgers, there'll be a current ledger that we write to. If curent ledger is closed(we shut down the service) or full a new ledger will be created and become the current ledger. There's a caveat that ledger trimming is performed on ledger level, e.g. it removes a whole ledger, and it **won't delete current ledger.** So in your case even if ledger retention is set to 10M and ledger size has exceeded, but it's the current ledger so it won't be removed. I think if you set `managedLedgerMaxSizePerLedgerMbytes` to lower value and produce messages to make topic write to a new ledger or just simple stop the service and start again and check stats again to activate the topic, you can observe the old ledger being deleted due to exceeded retention size limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-767744226
@sijie please correct me if I'm wrong
@lichuan6 ManagedLedger consists of many BookKeeper ledgers, there'll be a current ledger that we write to. If curent ledger is closed or full a new ledger will be created and become the current ledger. There's a caveat that ledger trimming is performed on ledger level, e.g. it removes a whole ledger, and it **won't delete current ledger.** So in your case even if ledger retention is set to 10M and ledger size has exceeded, but it's the current ledger so it won't be removed. I think if you set `managedLedgerMaxSizePerLedgerMbytes` to lower value and produce messages to make topic write to a new ledger, you can observe the old ledger being deleted due to exceeded retention size limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-788918972
Thanks @lichuan6, close this issue first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-768841443
@MarvinCai Thanks, one thing I want to add is the ledger been removed does not means the disk's resource will be released. It's related to the bookkeeper side. Bookkeeper will remove the data from the disk after the compaction task triggered or all of ledgers of an entry log been deleted. So @lichuan6 It's better to produce more data and make sure the ledger rollovered and wait for a while.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #9271: topic retention not work
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #9271:
URL: https://github.com/apache/pulsar/issues/9271#issuecomment-767744226
@sijie please correct me if I'm wrong
@lichuan6 ManagedLedger consists of many BookKeeper ledgers, there'll be a current ledger that we write to. If curent ledger is closed(we shut down the service) or full a new ledger will be created and become the current ledger. There's a caveat that ledger trimming is performed on ledger level, e.g. it removes a whole ledger, and it **won't touch current ledger.** So in your case even if ledger retention is set to 10M and ledger size has exceeded, but it's the current ledger so it won't be removed. I think if you set `managedLedgerMaxSizePerLedgerMbytes` to lower value and produce messages to make topic write to a new ledger or just simplelu stop the service and start again and check stats again to activate the topic, you can observe the old ledger being deleted due to exceeded retention size limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org