You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/26 18:43:26 UTC
[GitHub] [pulsar] ErikJansenIRefact opened a new issue #7372: Messages on topic are deleted on restart of standalone cluster
ErikJansenIRefact opened a new issue #7372:
URL: https://github.com/apache/pulsar/issues/7372
**Describe the bug**
Messages on a topic are deleted / not available anymore after standalone cluster shutdown and restart.
**To Reproduce**
Produce messages on a topic in a standalone cluster on a namespace with retention_period = -1.
Cancel the standalone cluster (Ctrl-C).
Restart the standalone cluster.
Browse pulsar manager and look at the storage of the topic. The pulsar manager reports 2 ledgers and total space used: 350Mb.
After a while (a few minutes) refresh the page. Pulsar manager reports the topic contains 0 Mb. One of the ledgers has disappeared.
**Expected behavior**
It was expected that messages are kept forever as the retention_period on the namespace is set to -1.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: IOS
- Pulsar 2.6.0
**Additional context**
Add any other context about the problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ErikJansenIRefact commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
ErikJansenIRefact commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-650965118
@sijie
The retentionPeriod on the namespace was set to -1 (infinite retention). However, this does not prevent that the data on the topic is removed. It seems that retentionSize should be set to -1 as well. So these settings should both be set to infinite. Only setting one of them isn't sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang closed issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
Jennifer88huang closed issue #7372:
URL: https://github.com/apache/pulsar/issues/7372
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ErikJansenIRefact commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
ErikJansenIRefact commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-685958856
Does make sense. But should be documented more clearly that this is the expected behaviour.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ErikJansenIRefact commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
ErikJansenIRefact commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-663452862
@sijie Any progression? Shouldn't this issue be labelled as type/bug?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-652839602
@ErikJansenIRefact this sounds like a bug. @codelipenghui can you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-650860281
@ErikJansenIRefact
By default, Pulsar doesn't configure any retention and `brokerDeleteInactiveTopicsEnabled` is enabled by default. So if a topic doesn't have any subscriptions and doesn't configure any retention period, the topic will be garbage collected by Pulsar broker due to its inactivity. The behavior has been documented in https://pulsar.apache.org/docs/en/2.6.0/admin-api-partitioned-topics/#create.
You can check the following settings:
```
# Enable the deletion of inactive topics
brokerDeleteInactiveTopicsEnabled=true
# How often to check for inactive topics
brokerDeleteInactiveTopicsFrequencySeconds=60
# Set the inactive topic delete mode. Default is delete_when_no_subscriptions
# 'delete_when_no_subscriptions' mode only delete the topic which has no subscriptions and no active producers
# 'delete_when_subscriptions_caught_up' mode only delete the topic that all subscriptions has no backlogs(caught up)
# and no active producers/consumers
brokerDeleteInactiveTopicsMode=delete_when_no_subscriptions
# Max duration of topic inactivity in seconds, default is not present
# If not present, 'brokerDeleteInactiveTopicsFrequencySeconds' will be used
# Topics that are inactive for longer than this value will be deleted
brokerDeleteInactiveTopicsMaxInactiveDurationSeconds=
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-669028474
If this makes sense to you I will close this issue and feel free to reopen it if you have any questions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ErikJansenIRefact commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
ErikJansenIRefact commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-650413163
It seems both retentionSize and retentionPeriod should be set to -1? Is this correct? Is this new as of release 2.6.0?
If so perhaps this should be more clearly documented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-717769466
@ErikJansenIRefact I've submitted a PR in #8397, could you please help review it? Any issue, feel free to comment, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #7372: Messages on topic are deleted on restart of standalone cluster
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #7372:
URL: https://github.com/apache/pulsar/issues/7372#issuecomment-666045385
> The retentionPeriod on the namespace was set to -1 (infinite retention). However, this does not prevent that the data on the topic is removed. It seems that retentionSize should be set to -1 as well. So these settings should both be set to infinite. Only setting one of them isn't sufficient.
Hi @ErikJansenIRefact. Yes, you are right.
There are two conditions for controlling the retention of a topic. Only both of them are specified as infinite, the topic will save forever. If there is a configuration that has satisfied with the delete condition, the topic will be deleted.
```
# Default message retention time
defaultRetentionTimeInMinutes=0
# Default retention size
defaultRetentionSizeInMB=0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org