You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/12 08:46:55 UTC
[GitHub] [pulsar] shibd opened a new issue, #15560: PIP-166: Function add `NONE` delivery semantics
shibd opened a new issue, #15560:
URL: https://github.com/apache/pulsar/issues/15560
## Motivation
Currently Function supports three delivery semantics, and also provides autoAck to control whether to automatically ack.
Because autoAck affects the delivery semantics of Function, it can be confusing for users to understand the relationship between these two parameters.
For example, when the user configures `Guarantees == ATMOST_ONCE` and `autoAck == false`, then the framework will not help the user to ack messages, and the processing semantics may become `ATLEAST_ONCE`.
The delivery semantics provided by Function should be clear. When the user sets the guarantees, the framework should ensure point-to-point semantic processing and cannot be affected by other parameters.
## Goal
Added `NONE` delivery semantics and delete `autoAck` config.
The original intention of `autoAck` semantics is that users want to control the timing of ack by themselves. When autoAck == false, the processing semantics provided by the framework should be invalid. Then we can add `NONE` processing semantics to replace the autoAck == false scenario.
When the user configuration `ProcessingGuarantees == NONE`, the framework does not help the user to do any ack operations, and the ack is left to the user to handle. In other cases, the framework guarantees processing semantics.
## API Changes
1. Add `NONE` type to ProcessingGuarantees
``` java
public enum ProcessingGuarantees {
ATLEAST_ONCE,
ATMOST_ONCE,
EFFECTIVELY_ONCE,
NONE
}
```
2. Delete autoAck config in FunctionConfig
``` java
public class FunctionConfig {
- private Boolean autoAck;
}
```
## Implementation
1. In `PulsarSinkAtLeastOnceProcessor` and `PulsarSinkEffectivelyOnceProcessor`, when `ProcessingGuarantees != NONE` can be ack.
<https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L274-L276>
2. When the delivery semantic is `ATMOST_ONCE`, the message will be acked immediately after receiving the message, no longer affected by the autoAck configuration.
https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L271-L276
3. When user call `record.ack()` in function, just `ProcessingGuarantees == NONE` can be work.
## Plan test
The main test and assert is that when ProcessingGuarantees == NONE, the function framework will not do any ack operations for the user.
## Compatibility
1. This change will invalidate the user's setting of autoAck, which should be explained in the documentation and provide parameter verification to remind the user.
2. Runtimes of other languages need to maintain consistent processing logic (python, go).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on issue #15560: PIP-166: Function add `MANUAL` delivery semantics
Posted by GitBox <gi...@apache.org>.
shibd commented on issue #15560:
URL: https://github.com/apache/pulsar/issues/15560#issuecomment-1149516406
> I have one last comment: "then delete it"
>
> We cannot "delete it", at least we can ignore it, otherwise when you upgrade Pulsar the function won't start. There are huge Pulsar multi-tenant clusters and we cannot force users to find all the instances of such functions and then upgrade the configuration.
Thank review. Agreed, I changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd closed issue #15560: PIP-166: Function add `MANUAL` delivery semantics
Posted by GitBox <gi...@apache.org>.
shibd closed issue #15560: PIP-166: Function add `MANUAL` delivery semantics
URL: https://github.com/apache/pulsar/issues/15560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #15560: PIP-166: Function add `MANUAL` delivery semantics
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #15560:
URL: https://github.com/apache/pulsar/issues/15560#issuecomment-1148615561
I have one last comment:
"then delete it"
We cannot "delete it", at least we can ignore it, otherwise when you upgrade Pulsar the function won't start.
There are huge Pulsar multi-tenant clusters and we cannot force users to find all the instances of such functions and then upgrade the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15560: PIP-166: Function add `MANUAL` delivery semantics
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15560:
URL: https://github.com/apache/pulsar/issues/15560#issuecomment-1197577810
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org