You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/10/28 08:05:55 UTC
[GitHub] [pulsar] Jennifer88huang opened a new pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Jennifer88huang opened a new pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397
Fixes #7372
### Motivation
The retention policy is not clearly documented.
### Modifications
Add detailed info on retention policy.
Add default value for "defaultRetentionTimeInMinutes"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718698312
@lhotari Thanks for your feedback.
I'm not sure if we use "Based on time and size limits", whether users are clear about the meaning it conveys.
@ErikJansenIRefact Do you have any opinion on it? Which is more clear to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718595249
@lhotari oops, it's my mistake. I'm sorry I forget to add the condition...
Thank you very much for your reminder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718055016
The code tells the truth:
https://github.com/apache/pulsar/blob/a8eaeb5eb6ddabcbc59030ea14207b43ca76237a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2091-L2097
The retention happens only when both conditions are met.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718390508
@ErikJansenIRefact
> Hi,
>
> In general good improvements in describing the behaviour of the retention settings.
> I would suggest however that you describe the behaviour when both settings (time and size) have a value > 0. Are the messages now removed if one of the thresholds exceeds or only if both of them are exceeded. So are the settings a "logical and" or a "logical or"?
>
> And is the 0-value treated as a special value? So, if size = 10Gb and time=0 are message now removed if size > 10Gb?
> And if size = 10Gb and time = 1d are messages now removed if either the size or the time is exceeded?
Good question. it's "logical or". However, if you set either to `0`, the retention is disabled.
@lhotari Thank you for helping answer the question, and provide the related doc.
I'll add more details here to make it clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718048764
> I would suggest however that you describe the behaviour when both settings (time and size) have a value > 0. Are the messages now removed if one of the thresholds exceeds or only if both of them are exceeded. So are the settings a "logical and" or a "logical or"?
good points. explaining this would be useful. I believe it's described somewhere. I'm actually unsure of the answer off the top of my head...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718048764
> I would suggest however that you describe the behaviour when both settings (time and size) have a value > 0. Are the messages now removed if one of the thresholds exceeds or only if both of them are exceeded. So are the settings a "logical and" or a "logical or"?
good points. explaining this would be useful. I believe it's described somewhere. It's a "logical or". Whatever limit is hit first will cause the data to not be retained (and thus a candidate for deletion).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718556253
> "Acknowledged messages or messages with no active subscription will not be retained"
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718055016
The code tells the truth:
https://github.com/apache/pulsar/blob/a8eaeb5eb6ddabcbc59030ea14207b43ca76237a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2075-L2101
The retention happens only when both conditions are met.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718055016
The code tells the truth:
https://github.com/apache/pulsar/blob/a8eaeb5eb6ddabcbc59030ea14207b43ca76237a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2075-L2101
The retention happens only when both conditions are met.
@Jennifer88huang please check if this is clearly documented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Huanli-Meng commented on a change in pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on a change in pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#discussion_r513337488
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
+
+The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
Review comment:
```suggestion
The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acknowledged by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
```
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
Review comment:
```suggestion
When you set a size limit of, for example, 10 GB, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit (10 GB). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
```
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
+
+The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
When a retention limit is exceeded, the oldest message is marked for deletion until the set of retained messages falls within the specified limits again.
### Defaults
-There are two configuration parameters that you can use to set [instance](reference-terminology.md#instance)-wide defaults for message retention: [`defaultRetentionTimeInMinutes=0`](reference-configuration.md#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB=0`](reference-configuration.md#broker-defaultRetentionSizeInMB).
+You can use the following two configuration parameters to set instance-wide defaults for message retention: `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB`. Both parameters are set to `0` by default.
Review comment:
change instance-wide to instance-level?
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
+
+The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
When a retention limit is exceeded, the oldest message is marked for deletion until the set of retained messages falls within the specified limits again.
### Defaults
-There are two configuration parameters that you can use to set [instance](reference-terminology.md#instance)-wide defaults for message retention: [`defaultRetentionTimeInMinutes=0`](reference-configuration.md#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB=0`](reference-configuration.md#broker-defaultRetentionSizeInMB).
+You can use the following two configuration parameters to set instance-wide defaults for message retention: `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB`. Both parameters are set to `0` by default.
-Both of these parameters are in the [`broker.conf`](reference-configuration.md#broker) configuration file.
+For more information of the two parameters, refer to the [`broker.conf`](reference-configuration.md#broker) configuration file.
### Set retention policy
-You can set a retention policy for a namespace by specifying the namespace as well as both a size limit *and* a time limit.
-
-#### pulsar-admin
-
-Use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-
-You must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to 0.
+You can set a retention policy for a namespace by specifying the namespace, a size limit and a time limit in `pulsar-admin`, REST API and Java.
-##### Examples
+<!--DOCUSAURUS_CODE_TABS-->
+<!--pulsar-admin-->
+You can use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-To set a size limit of 10 gigabytes and a time limit of 3 hours for the `my-tenant/my-ns` namespace:
+In the following example, size limit is set to 10 gigabytes and time limit is set to 3 hours for the `my-tenant/my-ns` namespace:
```shell
$ pulsar-admin namespaces set-retention my-tenant/my-ns \
--size 10G \
--time 3h
```
-To set retention where time limit is ignored and the size limit of 1 terabyte determines retention:
+In the following example, time is not limited and size limit is set to 1 terabyte. The size limit determines the retention result.
Review comment:
```suggestion
In the following example, the time is not limited while the size limit is set to 1 TB. The size limit determines the retention result.
```
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
+
+The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
When a retention limit is exceeded, the oldest message is marked for deletion until the set of retained messages falls within the specified limits again.
### Defaults
-There are two configuration parameters that you can use to set [instance](reference-terminology.md#instance)-wide defaults for message retention: [`defaultRetentionTimeInMinutes=0`](reference-configuration.md#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB=0`](reference-configuration.md#broker-defaultRetentionSizeInMB).
+You can use the following two configuration parameters to set instance-wide defaults for message retention: `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB`. Both parameters are set to `0` by default.
-Both of these parameters are in the [`broker.conf`](reference-configuration.md#broker) configuration file.
+For more information of the two parameters, refer to the [`broker.conf`](reference-configuration.md#broker) configuration file.
### Set retention policy
-You can set a retention policy for a namespace by specifying the namespace as well as both a size limit *and* a time limit.
-
-#### pulsar-admin
-
-Use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-
-You must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to 0.
+You can set a retention policy for a namespace by specifying the namespace, a size limit and a time limit in `pulsar-admin`, REST API and Java.
-##### Examples
+<!--DOCUSAURUS_CODE_TABS-->
+<!--pulsar-admin-->
+You can use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-To set a size limit of 10 gigabytes and a time limit of 3 hours for the `my-tenant/my-ns` namespace:
+In the following example, size limit is set to 10 gigabytes and time limit is set to 3 hours for the `my-tenant/my-ns` namespace:
```shell
$ pulsar-admin namespaces set-retention my-tenant/my-ns \
--size 10G \
--time 3h
```
-To set retention where time limit is ignored and the size limit of 1 terabyte determines retention:
+In the following example, time is not limited and size limit is set to 1 terabyte. The size limit determines the retention result.
```shell
$ pulsar-admin namespaces set-retention my-tenant/my-ns \
--size 1T \
--time -1
```
-To set retention where size limit is ignored and the time limit of 3 hours determines retention:
+In the following example, size is not limited and time limit is set to 3 hours. The time limit determines the retention result.
Review comment:
```suggestion
In the following example, the size is not limited and the time limit is set to 3 hours. The time limit determines the retention result.
```
##########
File path: site2/docs/cookbooks-retention-expiry.md
##########
@@ -27,90 +27,90 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
## Retention policies
-By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged on all subscriptions, at which point it will be marked for deletion. You can override this behavior and retain even messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
+By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
-Retention policies are particularly useful if you intend to exclusively use the Reader interface. Because the Reader interface does not use acknowledgements, messages will never exist within backlogs. Most realistic Reader-only use cases require that retention be configured.
+Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy you must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to `0`. Infinite retention can be achieved by setting both time and size limits to `-1`.
+When you set a retention policy, you must set **both** a *size limit* and a *time limit*.
-When you set a size limit of, say, 10 gigabytes, and the time limit to `-1` then acknowledged messages in all topics in the namespace will be retained until the size limit for the topic is reached; if you set a time limit of, say, 1 day, and the size limit to `-1` then acknowledged messages for all topics in the namespace will be retained for 24 hours.
+- If you do not want to limit the message by time, set the value of time limit to `-1`.
+- If you do not want to limit the message by size, set the value of size limit to `-1`.
+- If you need infinite retention, set the values of time limit and size limit to `-1`.
+- If you want to disable retention policy, set the values of time limit and size limit to `0`. Retention policy is disabled by default.
-The retention settings apply to all messages on topics that do not have any subscriptions, or if there are subscriptions, to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions -- these are instead controlled by the backlog quota (see below).
+When you set a size limit of, for example, 10 gigabytes, and set the time limit to `-1`, then the acknowledged messages in all topics in the namespace are retained until the topic size reaches the size limit(10 gigabytes). If you set a time limit of, for example, 1 day, and set the size limit to `-1`, then the acknowledged messages for all topics in the namespace are retained for 1 day.
+
+The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acked by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.
When a retention limit is exceeded, the oldest message is marked for deletion until the set of retained messages falls within the specified limits again.
### Defaults
-There are two configuration parameters that you can use to set [instance](reference-terminology.md#instance)-wide defaults for message retention: [`defaultRetentionTimeInMinutes=0`](reference-configuration.md#broker-defaultRetentionTimeInMinutes) and [`defaultRetentionSizeInMB=0`](reference-configuration.md#broker-defaultRetentionSizeInMB).
+You can use the following two configuration parameters to set instance-wide defaults for message retention: `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB`. Both parameters are set to `0` by default.
-Both of these parameters are in the [`broker.conf`](reference-configuration.md#broker) configuration file.
+For more information of the two parameters, refer to the [`broker.conf`](reference-configuration.md#broker) configuration file.
### Set retention policy
-You can set a retention policy for a namespace by specifying the namespace as well as both a size limit *and* a time limit.
-
-#### pulsar-admin
-
-Use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-
-You must set **both** a *size limit* and a *time limit*. In the case where you don't want to limit by either time or set, the value must be set to `-1`. Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged messages when either size or time limit is set to 0.
+You can set a retention policy for a namespace by specifying the namespace, a size limit and a time limit in `pulsar-admin`, REST API and Java.
-##### Examples
+<!--DOCUSAURUS_CODE_TABS-->
+<!--pulsar-admin-->
+You can use the [`set-retention`](reference-pulsar-admin.md#namespaces-set-retention) subcommand and specify a namespace, a size limit using the `-s`/`--size` flag, and a time limit using the `-t`/`--time` flag.
-To set a size limit of 10 gigabytes and a time limit of 3 hours for the `my-tenant/my-ns` namespace:
+In the following example, size limit is set to 10 gigabytes and time limit is set to 3 hours for the `my-tenant/my-ns` namespace:
Review comment:
```suggestion
In the following example, the size limit is set to 10 GB and time limit is set to 3 hours for the `my-tenant/my-ns` namespace:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718570487
@lhotari thanks for your quick response. Updated, PTAL again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718046439
> And is the 0-value treated as a special value?
@ErikJansenIRefact
Yes, `0` is a special value. Setting either limit to `0` will disable retention. That's the catch.
PR #8358 introduced new behavior that prevents setting just one of the values to 0 via the Admin API. To disable the retention policy, both values have to be set to 0 when using the Admin API.
The "setting either limit to 0 will disable retention" behavior still applies to existing configurations (after upgrading to 2.7.0 when it's out) and also for `defaultRetentionTimeInMinutes` and `defaultRetentionSizeInMB` settings. I wonder how to get this documented?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718500175
@Jennifer88huang just one minor comment about "Messages are deleted when either time or size reaches the limit". The might lead to misunderstanding that deletion applies for all messages when someone looks at this table and doesn't read the other parts where it's described. I'm not sure how to refer that this only applies to the messages that have been acknowledged or don't have an active subscription at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718578168
@Jennifer88huang I guess the "when either..." part is needed too. Something like "Acknowledged messages or messages with no active subscription will not be retained when either time or size reaches the limit."?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718048764
> I would suggest however that you describe the behaviour when both settings (time and size) have a value > 0. Are the messages now removed if one of the thresholds exceeds or only if both of them are exceeded. So are the settings a "logical and" or a "logical or"?
good points. explaining this would be useful. I believe it's described somewhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang merged pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang merged pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718486344
@ErikJansenIRefact I've listed the value settings in a table, is this way much clearer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718489666
@lhotari according to your comment,
- in 2.6.x, the setting is sth like this
- in 2.7, the setting is sth like this.
Do I grasp your meaning correctly? I've made adjustment in docs for both 2.7 and 2.6.1 accordingly (add the above table and examples). Any issue, feel free to comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718525725
@lhotari great, thanks for your reminder. How about replacing "Messages are deleted when either time or size reaches the limit" with one of the followings
- "Acknowledged messages or messages with no active subscription will be deleted"
- "Acknowledged messages or messages with no active subscription will not be retained"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718492964
> Do I grasp your meaning correctly? I've made adjustment in docs for both 2.7 and 2.6.1 accordingly (add the above table and examples). Any issue, feel free to comment.
Yes, perfect. Good work @Jennifer88huang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #8397: [Issue 7372][docs] Add detailed info for retention policy
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #8397:
URL: https://github.com/apache/pulsar/pull/8397#issuecomment-718613847
I read the document once more and it seems after all that it would be better to simplify the table. One suggestion is to replace "Acknowledged messages or messages with no active subscription will not be retained when either time or size reaches the limit." in the table with "Based on time and size limits" since it would be more consistent with the other rows in the same table.
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org