You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/14 13:45:41 UTC

[GitHub] [pulsar] tpiperatgod opened a new pull request, #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

tpiperatgod opened a new pull request, #18929:
URL: https://github.com/apache/pulsar/pull/18929

   <!--
   ### Contribution Checklist
     
     - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*. 
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   -->
   
   <!-- Either this PR fixes an issue, -->
   
   Fixes #18903 
   
   <!-- or this PR is one task of an issue -->
   
   Master Issue: #xyz
   
   ### Motivation
   
   Make pulsar python functions support to set `processingGuarantees` to `EFFECTIVELY_ONCE`
   
   ### Modifications
   
   <!-- Describe the modifications you've done. -->
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/tpiperatgod/pulsar/pull/6
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1447424313

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1462473304

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1429035380

   Thanks, @shibd, based on your comment, I think the mechanism of the effectively-once semantics should be as follows.
   
   The user needs to ensure that one source topic corresponds to one sink topic, at this time the principle of message delivery is: `source topic -> sink topic`
   If partition topic is enabled, it is necessary to ensure that the number of partitions of source topic and sink topic is the same, and ensure that the principle of message delivery follows: `source topic Pn -> sink topic Pn; source topic Pm -> sink topic Pm`
   
   Therefore, before using the effectively-once semantics, we should confirm the following points.
   1. deduplication is enabled
   2. function has only one source topic and one sink topic
   3. if partition topic is enabled, we need to make sure the number of partitions is the same
   
   Please feel free to correct me, and what do you think @freeznet @nlu90 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] freeznet commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "freeznet (via GitHub)" <gi...@apache.org>.
freeznet commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1445977320

   @shibd IIRC the python client do not support `PartitionsRoutingMode.CustomPartition` so it is hard to support [3] in python runtime unless the client supports it first. We could create an issue in `pulsar-client-python` and related pulsar function issue in `pulsar` repo as well. Once it is done, we could work on the partitioned topic support.
   
   As for this PR, it is okay for me to make the effectively_once work only with non-partitioned topics, WDYT @nlu90 @shibd @tpiperatgod ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1432788323

   OK, so how about this PR only supports `effectively-once` semantics for non-partitioned topics? or should I put it into the backlog?
   
   and we may need to tell users to enable `deduplication` through documentation to use the `effectively-once` semantics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1479197275

   This is fine by me, but as discussed in this PR, this semantic is not fully supported (for partitioned topics)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by github-actions.
github-actions[bot] commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1399384659

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1459131466

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "Anonymitaet (via GitHub)" <gi...@apache.org>.
Anonymitaet commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1480433806

   @tpiperatgod thanks for your feedback! 
   
   Double-check: should I make it more accurate as below?
   
   - [pulsar-admin reference](https://pulsar.apache.org/reference/#/next/pulsar-admin/functions?id=localrun)
   
   <img width="1714" alt="image" src="https://user-images.githubusercontent.com/50226895/227070377-5200049f-9dde-4903-82b7-038616513644.png">
   
   - [Pulsar Feature Matrix (Client + Function)](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1588411801&range=48:48)
   
   <img width="992" alt="image" src="https://user-images.githubusercontent.com/50226895/227070811-0faf4a6e-0f25-4b0c-a9d8-d7ca8d89582f.png">
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1446001823

   Hello @tpiperatgod 
   thank you very much for your contribution.
   Can you please raise a discussion on the community mailing list dev@pulsar.apache.org ?
   This will help the community to be informed of this important work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 merged pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 merged PR #18929:
URL: https://github.com/apache/pulsar/pull/18929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1476598196

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1427713593

   > @tpiperatgod Could you explain how these changes together ensure the `effectively_once` processing guarantee?
   
   Sorry for not finishing it before. It is currently available for review.
   
   this PR does the following
   1. maintain a `message id - producer obj` map during runtime
   2. when a message comes in, generate the `producer_name`, `producer_id` and the `sequence_id` based on the message information
   3. use `producer_id` to check if the corresponding producer obj exists: if it exists, set `self.producer` to that producer obj; otherwise, create a new producer obj using `producer_name`
   4. assign the `sequence_id` when sending the message
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1449508096

   > Hello @tpiperatgod thank you very much for your contribution. Can you please raise a discussion on the community mailing list [dev@pulsar.apache.org](mailto:dev@pulsar.apache.org) ? This will help the community to be informed of this important work
   
   please follow up this issue via the link: https://lists.apache.org/thread/qvm955jsmskqx6lhg2mmzft4745lj7th


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] tpiperatgod commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "tpiperatgod (via GitHub)" <gi...@apache.org>.
tpiperatgod commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1446112003

   > Hello @tpiperatgod thank you very much for your contribution. Can you please raise a discussion on the community mailing list [dev@pulsar.apache.org](mailto:dev@pulsar.apache.org) ? This will help the community to be informed of this important work
   
   sure, I will.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "Anonymitaet (via GitHub)" <gi...@apache.org>.
Anonymitaet commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1477383424

   Hi @tpiperatgod @freeznet @nlu90 this PR is labeled with `doc-required`. I check the relevant docs and have some questions below. Can you please double-check? Thank you!
   
   1.  Doc for `pulsar-admin` reference is done.
   
       "Python" has been added for `localrun`, `create`, `update` commands, so we do not need to do anything more for this reference doc.
     
     <img width="1919" alt="image" src="https://user-images.githubusercontent.com/50226895/226540235-14ee72a6-eb0e-418b-bc60-60406094bb39.png">
   
   2. [Pulsar Feature Matrix (Client + Function)](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1588411801&range=48:48) needs to be updated.
   
       Should the Python column of lines 48, 117, 177 be changed as below?
   
   <img width="996" alt="image" src="https://user-images.githubusercontent.com/50226895/226541248-9b075b5e-fb5b-41a1-8e1a-f8f3bd50a389.png">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] freeznet commented on a diff in pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "freeznet (via GitHub)" <gi...@apache.org>.
freeznet commented on code in PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#discussion_r1118486643


##########
pulsar-functions/instance/src/main/python/python_instance.py:
##########
@@ -302,14 +310,27 @@ def done_producing(self, consumer, orig_message, topic, result, sent_message):
       # If producer fails send output then send neg ack for input message back to broker
       consumer.negative_acknowledge(orig_message)
 
-
   def process_result(self, output, msg):
-    if output is not None and self.instance_config.function_details.sink.topic != None and \
+    if output is not None and self.instance_config.function_details.sink.topic is not None and \
             len(self.instance_config.function_details.sink.topic) > 0:
       if self.output_serde is None:
         self.setup_output_serde()
+      if self.effectively_once:
+        if self.contextimpl.get_message_partition_index() is None or \
+                self.contextimpl.get_message_partition_index() >= 0:
+          Log.error("Partitioned topic is not available in effectively_once mode.")
+          return

Review Comment:
   Should raise an error here instead of `return` nothing.



##########
pulsar-functions/instance/src/main/python/python_instance.py:
##########
@@ -475,6 +509,22 @@ def get_function_status(self):
     status.lastInvocationTime = int(last_invocation) if sys.version_info.major >= 3 else long(last_invocation)
     return status
 
+  def can_enable_effectively_once(self):
+    """
+    The prerequisites for enabling effective-once semantics are as follows.
+    1. deduplication is enabled
+    2. set ProcessingGuarantees to EFFECTIVELY_ONCE
+    3. the function has only one source topic and one sink topic
+    4. (unsupported yet) if partitioned topic is enabled, ensure that the number of partitions
+      (of both source and sink topics) is the same
+    """
+    if self.instance_config.function_details.processingGuarantees == \
+            Function_pb2.ProcessingGuarantees.Value('EFFECTIVELY_ONCE') and \

Review Comment:
   Could you please add some log output here when `processingGuarantees` is set to `EFFECTIVELY_ONCE`, but other requirements do not meet?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] shibd commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1433399875

   > OK, so how about this PR only supports `effectively-once` semantics for non-partitioned topics? or should I put it into the backlog?
   > 
   > and we may need to tell users to enable `deduplication` through documentation to use the `effectively-once` semantics.
   
   We can discuss how to achieve [3] first. @nlu90 @freeznet Can you take a look at it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1358722665

   # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18929?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#18929](https://codecov.io/gh/apache/pulsar/pull/18929?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3c668c6) into [master](https://codecov.io/gh/apache/pulsar/commit/050b3106ad07070c0fe3c26e873ad57d96d3f250?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (050b310) will **decrease** coverage by `10.52%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pulsar/pull/18929/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pulsar/pull/18929?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #18929       +/-   ##
   =============================================
   - Coverage     47.35%   36.82%   -10.53%     
   + Complexity     9384     1958     -7426     
   =============================================
     Files           623      209      -414     
     Lines         59104    14402    -44702     
     Branches       6146     1569     -4577     
   =============================================
   - Hits          27987     5304    -22683     
   + Misses        28100     8524    -19576     
   + Partials       3017      574     -2443     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `36.82% <ø> (-10.53%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18929?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...he/pulsar/client/impl/PartitionedProducerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1BhcnRpdGlvbmVkUHJvZHVjZXJJbXBsLmphdmE=) | `30.34% <0.00%> (-5.13%)` | :arrow_down: |
   | [...va/org/apache/pulsar/client/impl/ProducerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1Byb2R1Y2VySW1wbC5qYXZh) | `15.66% <0.00%> (-1.17%)` | :arrow_down: |
   | [.../pulsar/client/impl/ProducerStatsRecorderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1Byb2R1Y2VyU3RhdHNSZWNvcmRlckltcGwuamF2YQ==) | `84.04% <0.00%> (-0.62%)` | :arrow_down: |
   | [...va/org/apache/pulsar/broker/service/ServerCnx.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1NlcnZlckNueC5qYXZh) | | |
   | [.../pulsar/broker/service/persistent/SystemTopic.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvU3lzdGVtVG9waWMuamF2YQ==) | | |
   | [...sar/broker/protocol/ProtocolHandlerDefinition.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9wcm90b2NvbC9Qcm90b2NvbEhhbmRsZXJEZWZpbml0aW9uLmphdmE=) | | |
   | [...tats/prometheus/metrics/PrometheusStatsLogger.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9wcm9tZXRoZXVzL21ldHJpY3MvUHJvbWV0aGV1c1N0YXRzTG9nZ2VyLmphdmE=) | | |
   | [...ion/buffer/metadata/TransactionBufferSnapshot.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvbWV0YWRhdGEvVHJhbnNhY3Rpb25CdWZmZXJTbmFwc2hvdC5qYXZh) | | |
   | [...roker/admin/impl/OffloaderObjectsScannerUtils.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL09mZmxvYWRlck9iamVjdHNTY2FubmVyVXRpbHMuamF2YQ==) | | |
   | [...ulsar/broker/intercept/BrokerInterceptorUtils.java](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9pbnRlcmNlcHQvQnJva2VySW50ZXJjZXB0b3JVdGlscy5qYXZh) | | |
   | ... and [409 more](https://codecov.io/gh/apache/pulsar/pull/18929/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] shibd commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1432356247

   Hi, 
   
   > Thanks, @shibd, based on your comment, I think the mechanism of the effectively-once semantics should be as follows.
   > 
   > The user needs to ensure that one source topic corresponds to one sink topic, at this time the principle of message delivery is: `source topic -> sink topic` If partition topic is enabled, it is necessary to ensure that the number of partitions of source topic and sink topic is the same, and ensure that the principle of message delivery follows: `source topic Pn -> sink topic Pn; source topic Pm -> sink topic Pm`
   > 
   > Therefore, before using the effectively-once semantics, we should confirm the following points.
   > 
   > 1. deduplication is enabled
   > 2. function has only one source topic and one sink topic
   > 3. if partition topic is enabled, we need to make sure the number of partitions is the same
   > 
   > Please feel free to correct me, and what do you think @freeznet @nlu90
   
   Hi, I agree 1, 2.
   
   For 3, I think even if the source topic and sink topic have the same number of partitions, the function framework also does not guarantee that the data of `source topic [Pn]` will be sent to `sink topic [Pn]`
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 closed pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 closed pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function 
URL: https://github.com/apache/pulsar/pull/18929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nlu90 commented on pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #18929:
URL: https://github.com/apache/pulsar/pull/18929#issuecomment-1474110783

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org