You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "shibd (via GitHub)" <gi...@apache.org> on 2024/11/06 11:08:37 UTC
[PR] Fix retention and backlog quota diagram [pulsar-site]
shibd opened a new pull request, #973:
URL: https://github.com/apache/pulsar-site/pull/973
The current retention and backlog quota diagram does not match the actual implementation.
Refer to issue: https://github.com/apache/pulsar/issues/22473#issuecomment-2426787328
Discuss thread: https://lists.apache.org/thread/g3hc3o0kmn1yvcfdj6xnlbq4vsdskx9o
### Preview
Retention policy
<img width="1293" alt="image" src="https://github.com/user-attachments/assets/691c196c-e746-4fda-94fb-0fc3abfd59e2">
Backlog quotas
<img width="1782" alt="image" src="https://github.com/user-attachments/assets/8a4bc6ba-d789-43d7-9261-1431948fd26f">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd merged PR #973:
URL: https://github.com/apache/pulsar-site/pull/973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1830874828
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
why are you switching to use .png files? How are these files created? What is the original source of the diagram if someone else would like to edit the diagram?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1832436352
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
Of course, but should we upload it directly to this repository? Each file is about 3MB, and I'm worried it might get too large in the future. Can we put it in SVN instead? For example: https://dist.apache.org/repos/dist/release/pulsar/pulsar-site/orignal-diagram
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1830974747
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
Where are the sources for these diagrams?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1831361477
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
Would it be possible to share the original OmniGraffle files? It would be useful if anyone could be able to maintain the diagrams that we have in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1832445445
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
just wondering why isn't the file `retention.svg` simply updated? If this diagram update applies to all versions of Pulsar, there shouldn't be a need to use a new file name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1831011852
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
I recreated it locally using OmniGraffle, and it supports generating SVG format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1832611890
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
Since I haven't modified the references for older versions, like 2.10, I've kept the original files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1833631573
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
@lhotari WHYT?
Do we need to update all versions of the documentation to point to the new diagrams? This way, I can remove the original diagrams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1830902484
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
I didn't notice that, thanks for reminding me. I've regenerated the file in SVG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1832623844
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
that makes sense. I push to ./diagrams directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1830990691
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
How did you regenerate the diagram?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1832442986
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.
The diagram below illustrates the concept of message retention.
-
+
Review Comment:
I don't think that 3MB is much in repository size. How many files are there for the diagram files?
Let's keep them in pulsar-site git repository, for example in `diagrams` sub directory. That would get ignored in the Docusaurus build so the website size wouldn't increase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix retention and backlog quota diagram [pulsar-site]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #973:
URL: https://github.com/apache/pulsar-site/pull/973#discussion_r1830872585
##########
docs/cookbooks-retention-expiry.md:
##########
@@ -205,7 +205,7 @@ The following retention policies are available:
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |
> #### Beware the distinction between retention policy types
-> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.
+> As you may have noticed, the scope of retention includes the backlog, so retention must be greater than the backlog quota; otherwise, Pulsar will throw an error when set both.
Review Comment:
Fix grammar in this sentence. "when set both" is not correct. "As you may have noticed" could be omitted. Suggestion:
```
:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org