You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "KevinLiLu (via GitHub)" <gi...@apache.org> on 2023/10/31 22:06:33 UTC

[PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

KevinLiLu opened a new pull request, #21490:
URL: https://github.com/apache/pulsar/pull/21490

   
   <!--
   ### Contribution Checklist
     
     - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*. 
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   -->
   
   <!-- If the PR belongs to a PIP, please add the PIP link here -->
   
   PIP-315: Configurable max delay limit for delayed delivery
   
   <!-- Details of when a PIP is required and how the PIP process work, please see: https://github.com/apache/pulsar/blob/master/pip/README.md -->
   
   ### Motivation
   
   <!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
   
   ### Modifications
   
   <!-- Describe the modifications you've done. -->
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: <!-- ENTER URL HERE -->
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "315157973 (via GitHub)" <gi...@apache.org>.
315157973 commented on code in PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#discussion_r1386296475


##########
pip/pip_315.md:
##########
@@ -0,0 +1,142 @@
+# PIP-315: Configurable max delay limit for delayed delivery
+
+# Background knowledge
+Delayed message delivery is an important feature which allows a producer to specify that a message should be delivered/consumed at a later time. Currently the broker will save a delayed message without any check. The message's `deliverAt` time is checked when the broker dispatches messages to the Consumer. If a message has a `deliverAt` time, then it is added to the `DelayedDeliveryTracker` and will be delivered later when eligible.
+
+Delayed message delivery is only available for persistent topics, and shared/key-shared subscription types.
+
+# Motivation
+Currently there is no max delay limit so a producer can specify any delay when publishing a message.
+
+This poses a few challenges:
+1. Producer may miscalculate/misconfigure a very large delay (ex. 1,000 day instead of 100 day delay)
+2. Pulsar administrators may want to limit the max allowed delay since unacked messages (ex. messages with a large delay) will be stored forever (unless TTL is configured)
+3. The configured delay may be greater than the configured TTL which means the delayed message may be deleted before the `deliverAt` time (before the consumer can process it)
+
+# Goals
+The purpose of this PIP is to introduce an optional configuration to limit the max allowed delay for delayed delivery.
+
+## In Scope
+- Add broker configuration to limit the max allowed delay for delayed delivery
+- Configurable at broker/topic/namespace-level
+
+## Out of Scope
+
+# High Level Design
+We will add a configuration `maxDeliveryDelayInSeconds` and if configured, the broker will check incoming delayed messages to see if the message's `deliverAt` time exceeds the configured limit. If it exceeds the limit, the broker will send an error back to the Producer.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Broker Changes
+A new `maxDeliveryDelayInSeconds` config will be added to the broker which is initially defaulted to 0 (disabled). The default (disabled) behavior will match the current delayed delivery behavior (no limit on delivery delay).
+```
+# broker.conf
+delayedDeliveryMaxDeliveryDelayInSeconds=0
+```
+
+This field will also be added to the existing `DelayedDeliveryPolicies` interface to support topic & namespace-level configuration:
+```java
+public interface DelayedDeliveryPolicies {
+    long getMaxDeliveryDelayInSeconds();
+}
+```
+
+The max delivery delay check will occur in the broker's `Producer` class inside of `checkAndStartPublish` (same place as other checks such as `isEncryptionEnabled`).
+
+We will give a `ServerError.NotAllowedError` error if all of the following are true:
+1. Sending to a persistent topic
+2. Topic has `delayedDeliveryEnabled=true`
+3. `MessageMetadata` `deliver_at_time` has been specified
+4. Topic has `>0` value for `maxDeliveryDelayInSeconds`
+5. `deliver_at_time` > `System.currentTimeMillis() + maxDeliveryDelayInSeconds * 1000`
+
+```java
+// In org.apache.pulsar.broker.service.Producer#checkAndStartPublish
+if (topic.isPersistent()) {
+    PersistentTopic pTopic = (PersistentTopic) topic;
+    if (pTopic.isDelayedDeliveryEnabled()) {
+        headersAndPayload.markReaderIndex();
+        MessageMetadata msgMetadata = Commands.parseMessageMetadata(headersAndPayload);
+        headersAndPayload.resetReaderIndex();
+        if (msgMetadata.hasDeliverAtTime()) {
+            long deliverAtTime = msgMetadata.getDeliverAtTime();
+            long maxDeliveryDelayInSeconds = pTopic.getMaxDeliveryDelayInSeconds();
+            if (maxDeliveryDelayInSeconds > 0
+                    && deliverAtTime > System.currentTimeMillis() + maxDeliveryDelayInSeconds * 1000) {

Review Comment:
   This seems like a good proposal. I have some comments
   1. Now the delay time accuracy reaches millisecond level, and the unit of the maximum delay threshold should also be consistent.
   2. Each message will be judged, so we should try to reduce unnecessary repeated calculations, such as `maxDeliveryDelayInSeconds` * 1000



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "KevinLiLu (via GitHub)" <gi...@apache.org>.
KevinLiLu commented on code in PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#discussion_r1391871992


##########
pip/pip_315.md:
##########
@@ -0,0 +1,142 @@
+# PIP-315: Configurable max delay limit for delayed delivery
+
+# Background knowledge
+Delayed message delivery is an important feature which allows a producer to specify that a message should be delivered/consumed at a later time. Currently the broker will save a delayed message without any check. The message's `deliverAt` time is checked when the broker dispatches messages to the Consumer. If a message has a `deliverAt` time, then it is added to the `DelayedDeliveryTracker` and will be delivered later when eligible.
+
+Delayed message delivery is only available for persistent topics, and shared/key-shared subscription types.
+
+# Motivation
+Currently there is no max delay limit so a producer can specify any delay when publishing a message.
+
+This poses a few challenges:
+1. Producer may miscalculate/misconfigure a very large delay (ex. 1,000 day instead of 100 day delay)
+2. Pulsar administrators may want to limit the max allowed delay since unacked messages (ex. messages with a large delay) will be stored forever (unless TTL is configured)
+3. The configured delay may be greater than the configured TTL which means the delayed message may be deleted before the `deliverAt` time (before the consumer can process it)
+
+# Goals
+The purpose of this PIP is to introduce an optional configuration to limit the max allowed delay for delayed delivery.
+
+## In Scope
+- Add broker configuration to limit the max allowed delay for delayed delivery
+- Configurable at broker/topic/namespace-level
+
+## Out of Scope
+
+# High Level Design
+We will add a configuration `maxDeliveryDelayInMillis` and if configured, the broker will check incoming delayed messages to see if the message's `deliverAt` time exceeds the configured limit. If it exceeds the limit, the broker will send an error back to the Producer.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Broker Changes
+A new `maxDeliveryDelayInMillis` config will be added to the broker which is initially defaulted to 0 (disabled). The default (disabled) behavior will match the current delayed delivery behavior (no limit on delivery delay).
+```
+# broker.conf
+delayedDeliveryMaxDeliveryDelayInMillis=0
+```
+
+This field will also be added to the existing `DelayedDeliveryPolicies` interface to support topic & namespace-level configuration:
+```java
+public interface DelayedDeliveryPolicies {
+    long getMaxDeliveryDelayInMillis();
+}
+```
+
+The max delivery delay check will occur in the broker's `Producer` class inside of `checkAndStartPublish` (same place as other checks such as `isEncryptionEnabled`).
+
+We will give a `ServerError.NotAllowedError` error if all of the following are true:
+1. Sending to a persistent topic
+2. Topic has `delayedDeliveryEnabled=true`
+3. `MessageMetadata` `deliver_at_time` has been specified
+4. Topic has `>0` value for `maxDeliveryDelayInMillis`
+5. `deliver_at_time` > `System.currentTimeMillis() + maxDeliveryDelayInMillis`

Review Comment:
   @coderzc Good idea. I have updated the PIP. Thanks for the suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "KevinLiLu (via GitHub)" <gi...@apache.org>.
KevinLiLu commented on code in PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#discussion_r1389975507


##########
pip/pip_315.md:
##########
@@ -0,0 +1,142 @@
+# PIP-315: Configurable max delay limit for delayed delivery
+
+# Background knowledge
+Delayed message delivery is an important feature which allows a producer to specify that a message should be delivered/consumed at a later time. Currently the broker will save a delayed message without any check. The message's `deliverAt` time is checked when the broker dispatches messages to the Consumer. If a message has a `deliverAt` time, then it is added to the `DelayedDeliveryTracker` and will be delivered later when eligible.
+
+Delayed message delivery is only available for persistent topics, and shared/key-shared subscription types.
+
+# Motivation
+Currently there is no max delay limit so a producer can specify any delay when publishing a message.
+
+This poses a few challenges:
+1. Producer may miscalculate/misconfigure a very large delay (ex. 1,000 day instead of 100 day delay)
+2. Pulsar administrators may want to limit the max allowed delay since unacked messages (ex. messages with a large delay) will be stored forever (unless TTL is configured)
+3. The configured delay may be greater than the configured TTL which means the delayed message may be deleted before the `deliverAt` time (before the consumer can process it)
+
+# Goals
+The purpose of this PIP is to introduce an optional configuration to limit the max allowed delay for delayed delivery.
+
+## In Scope
+- Add broker configuration to limit the max allowed delay for delayed delivery
+- Configurable at broker/topic/namespace-level
+
+## Out of Scope
+
+# High Level Design
+We will add a configuration `maxDeliveryDelayInSeconds` and if configured, the broker will check incoming delayed messages to see if the message's `deliverAt` time exceeds the configured limit. If it exceeds the limit, the broker will send an error back to the Producer.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Broker Changes
+A new `maxDeliveryDelayInSeconds` config will be added to the broker which is initially defaulted to 0 (disabled). The default (disabled) behavior will match the current delayed delivery behavior (no limit on delivery delay).
+```
+# broker.conf
+delayedDeliveryMaxDeliveryDelayInSeconds=0
+```
+
+This field will also be added to the existing `DelayedDeliveryPolicies` interface to support topic & namespace-level configuration:
+```java
+public interface DelayedDeliveryPolicies {
+    long getMaxDeliveryDelayInSeconds();
+}
+```
+
+The max delivery delay check will occur in the broker's `Producer` class inside of `checkAndStartPublish` (same place as other checks such as `isEncryptionEnabled`).
+
+We will give a `ServerError.NotAllowedError` error if all of the following are true:
+1. Sending to a persistent topic
+2. Topic has `delayedDeliveryEnabled=true`
+3. `MessageMetadata` `deliver_at_time` has been specified
+4. Topic has `>0` value for `maxDeliveryDelayInSeconds`
+5. `deliver_at_time` > `System.currentTimeMillis() + maxDeliveryDelayInSeconds * 1000`
+
+```java
+// In org.apache.pulsar.broker.service.Producer#checkAndStartPublish
+if (topic.isPersistent()) {
+    PersistentTopic pTopic = (PersistentTopic) topic;
+    if (pTopic.isDelayedDeliveryEnabled()) {
+        headersAndPayload.markReaderIndex();
+        MessageMetadata msgMetadata = Commands.parseMessageMetadata(headersAndPayload);
+        headersAndPayload.resetReaderIndex();
+        if (msgMetadata.hasDeliverAtTime()) {
+            long deliverAtTime = msgMetadata.getDeliverAtTime();
+            long maxDeliveryDelayInSeconds = pTopic.getMaxDeliveryDelayInSeconds();
+            if (maxDeliveryDelayInSeconds > 0
+                    && deliverAtTime > System.currentTimeMillis() + maxDeliveryDelayInSeconds * 1000) {

Review Comment:
   @315157973 Thanks for the review. You bring up a good point about the extra overhead per message when using seconds. I have updated the PIP to use milliseconds now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "KevinLiLu (via GitHub)" <gi...@apache.org>.
KevinLiLu commented on PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#issuecomment-1811779509

   Hi @315157973 @coderzc, I have started the vote thread and added the link.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #21490:
URL: https://github.com/apache/pulsar/pull/21490


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "KevinLiLu (via GitHub)" <gi...@apache.org>.
KevinLiLu commented on PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#issuecomment-1791369058

   Hi @codelipenghui @jiazhai @merlimat @coderzc @wolfstudy @315157973, I saw you worked on delayed delivery code in the past. Would you mind providing some thoughts on this PIP? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "315157973 (via GitHub)" <gi...@apache.org>.
315157973 commented on PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#issuecomment-1809778048

   This proposal needs to be voted in dev mail list at first
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#issuecomment-1788103353

   @KevinLiLu Please add the following content to your PR description and select a checkbox:
   ```
   - [ ] `doc` <!-- Your PR contains doc changes -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "KevinLiLu (via GitHub)" <gi...@apache.org>.
KevinLiLu commented on PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#issuecomment-1799408817

   Hi @lhotari @tisonkun @mattisonchao @rdhabalia @poorbarcode @Technoboy- , could you review this PIP when you get a chance? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][pip] PIP-315: Configurable max delay limit for delayed delivery [pulsar]

Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #21490:
URL: https://github.com/apache/pulsar/pull/21490#discussion_r1390696901


##########
pip/pip_315.md:
##########
@@ -0,0 +1,142 @@
+# PIP-315: Configurable max delay limit for delayed delivery
+
+# Background knowledge
+Delayed message delivery is an important feature which allows a producer to specify that a message should be delivered/consumed at a later time. Currently the broker will save a delayed message without any check. The message's `deliverAt` time is checked when the broker dispatches messages to the Consumer. If a message has a `deliverAt` time, then it is added to the `DelayedDeliveryTracker` and will be delivered later when eligible.
+
+Delayed message delivery is only available for persistent topics, and shared/key-shared subscription types.
+
+# Motivation
+Currently there is no max delay limit so a producer can specify any delay when publishing a message.
+
+This poses a few challenges:
+1. Producer may miscalculate/misconfigure a very large delay (ex. 1,000 day instead of 100 day delay)
+2. Pulsar administrators may want to limit the max allowed delay since unacked messages (ex. messages with a large delay) will be stored forever (unless TTL is configured)
+3. The configured delay may be greater than the configured TTL which means the delayed message may be deleted before the `deliverAt` time (before the consumer can process it)
+
+# Goals
+The purpose of this PIP is to introduce an optional configuration to limit the max allowed delay for delayed delivery.
+
+## In Scope
+- Add broker configuration to limit the max allowed delay for delayed delivery
+- Configurable at broker/topic/namespace-level
+
+## Out of Scope
+
+# High Level Design
+We will add a configuration `maxDeliveryDelayInMillis` and if configured, the broker will check incoming delayed messages to see if the message's `deliverAt` time exceeds the configured limit. If it exceeds the limit, the broker will send an error back to the Producer.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Broker Changes
+A new `maxDeliveryDelayInMillis` config will be added to the broker which is initially defaulted to 0 (disabled). The default (disabled) behavior will match the current delayed delivery behavior (no limit on delivery delay).
+```
+# broker.conf
+delayedDeliveryMaxDeliveryDelayInMillis=0
+```
+
+This field will also be added to the existing `DelayedDeliveryPolicies` interface to support topic & namespace-level configuration:
+```java
+public interface DelayedDeliveryPolicies {
+    long getMaxDeliveryDelayInMillis();
+}
+```
+
+The max delivery delay check will occur in the broker's `Producer` class inside of `checkAndStartPublish` (same place as other checks such as `isEncryptionEnabled`).
+
+We will give a `ServerError.NotAllowedError` error if all of the following are true:
+1. Sending to a persistent topic
+2. Topic has `delayedDeliveryEnabled=true`
+3. `MessageMetadata` `deliver_at_time` has been specified
+4. Topic has `>0` value for `maxDeliveryDelayInMillis`
+5. `deliver_at_time` > `System.currentTimeMillis() + maxDeliveryDelayInMillis`

Review Comment:
   Can we use `publishTime - deliverAtTime > maxDeliveryDelayInMillis` instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org