You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "nlu90 (via GitHub)" <gi...@apache.org> on 2023/02/07 00:13:27 UTC
[GitHub] [pulsar] nlu90 opened a new pull request, #17214: [WIP] configure whether function consumer should skip to latest
nlu90 opened a new pull request, #17214:
URL: https://github.com/apache/pulsar/pull/17214
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
Fixes #<xyz>
*(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>` to link to the master issue.)*
Master Issue: #<xyz>
### Motivation
In certain failure cases, the function needs to skip all the content between the last successfully Acked message and the latest message in the topic in order for quick recovery.
### Modifications
1. Providing a boolean config for function submission cmd
2. PulsarSource will call `consumer.seek(MessageId.latest)` if the skip flag is set
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [X] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1409108617
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on a diff in pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on code in PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#discussion_r1119307691
##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -165,6 +165,7 @@ message SourceSpec {
bool cleanupSubscription = 11;
SubscriptionPosition subscriptionPosition = 12;
uint64 negativeAckRedeliveryDelayMs = 13;
+ bool skipToLatest = 14;
Review Comment:
@michaeljmarshall As we discussed via slack, I'll email the mailing list about the function proto change.
But it should be backward compatible and won't cause any issue for previous users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 merged pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 merged PR #17214:
URL: https://github.com/apache/pulsar/pull/17214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1306521733
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1347644168
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liudezhi2098 commented on pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by GitBox <gi...@apache.org>.
liudezhi2098 commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1223376135
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "momo-jun (via GitHub)" <gi...@apache.org>.
momo-jun commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1447488826
@nlu90 shall we add `skipToLatest` to [here](https://pulsar.apache.org/docs/next/functions-cli/#consumerconfig)? Maybe @freeznet @liudezhi2098 can also advise.
BTW, the pulsar docs have been moved to the [site repo](https://github.com/apache/pulsar-site/tree/main/docs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui closed pull request #17214: [WIP] configure whether function consumer should skip to latest
URL: https://github.com/apache/pulsar/pull/17214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1420053717
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#17214](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a94a2b2) into [master](https://codecov.io/gh/apache/pulsar/commit/644be5fdd6d080accffd72229b2e21e35a27d722?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (644be5f) will **increase** coverage by `1.09%`.
> The diff coverage is `35.71%`.
[](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #17214 +/- ##
============================================
+ Coverage 32.42% 33.51% +1.09%
- Complexity 6347 6363 +16
============================================
Files 1644 1659 +15
Lines 123712 134220 +10508
Branches 13486 15740 +2254
============================================
+ Hits 40109 44985 +4876
- Misses 77694 82776 +5082
- Partials 5909 6459 +550
```
| Flag | Coverage Δ | |
|---|---|---|
| inttests | `26.39% <0.00%> (+1.45%)` | :arrow_up: |
| systests | `26.74% <35.71%> (+1.08%)` | :arrow_up: |
| unittests | `17.56% <0.00%> (+0.02%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...java/org/apache/pulsar/admin/cli/CmdFunctions.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2FkbWluL2NsaS9DbWRGdW5jdGlvbnMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ar/functions/source/MultiConsumerPulsarSource.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy9pbnN0YW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Z1bmN0aW9ucy9zb3VyY2UvTXVsdGlDb25zdW1lclB1bHNhclNvdXJjZS5qYXZh) | `86.84% <0.00%> (-4.83%)` | :arrow_down: |
| [...r/functions/source/SingleConsumerPulsarSource.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy9pbnN0YW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Z1bmN0aW9ucy9zb3VyY2UvU2luZ2xlQ29uc3VtZXJQdWxzYXJTb3VyY2UuamF2YQ==) | `76.66% <0.00%> (-12.62%)` | :arrow_down: |
| [...he/pulsar/functions/utils/FunctionConfigUtils.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy91dGlscy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Z1bmN0aW9ucy91dGlscy9GdW5jdGlvbkNvbmZpZ1V0aWxzLmphdmE=) | `40.24% <33.33%> (-0.04%)` | :arrow_down: |
| [...apache/pulsar/common/functions/FunctionConfig.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC1hZG1pbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jb21tb24vZnVuY3Rpb25zL0Z1bmN0aW9uQ29uZmlnLmphdmE=) | `96.42% <100.00%> (+0.06%)` | :arrow_up: |
| [...ulsar/functions/instance/JavaInstanceRunnable.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy9pbnN0YW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Z1bmN0aW9ucy9pbnN0YW5jZS9KYXZhSW5zdGFuY2VSdW5uYWJsZS5qYXZh) | `54.43% <100.00%> (+0.18%)` | :arrow_up: |
| [...he/pulsar/functions/source/PulsarSourceConfig.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy9pbnN0YW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Z1bmN0aW9ucy9zb3VyY2UvUHVsc2FyU291cmNlQ29uZmlnLmphdmE=) | `81.81% <100.00%> (+1.81%)` | :arrow_up: |
| [...e/pulsar/client/impl/auth/AuthenticationToken.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL2F1dGgvQXV0aGVudGljYXRpb25Ub2tlbi5qYXZh) | `37.14% <0.00%> (-14.29%)` | :arrow_down: |
| [...pache/pulsar/broker/web/ResponseHandlerFilter.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci93ZWIvUmVzcG9uc2VIYW5kbGVyRmlsdGVyLmphdmE=) | `27.77% <0.00%> (-12.77%)` | :arrow_down: |
| [.../apache/pulsar/metadata/impl/ZKSessionWatcher.java](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvaW1wbC9aS1Nlc3Npb25XYXRjaGVyLmphdmE=) | `72.46% <0.00%> (-10.15%)` | :arrow_down: |
| ... and [130 more](https://codecov.io/gh/apache/pulsar/pull/17214?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "momo-jun (via GitHub)" <gi...@apache.org>.
momo-jun commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1445773825
Hi @nlu90, I'm checking in to follow up with the doc updates. Did you have any plans to update the docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liudezhi2098 commented on pull request #17214: [WIP] configure whether function consumer should skip to latest
Posted by GitBox <gi...@apache.org>.
liudezhi2098 commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1311323509
@nlu90 help handle Conflicting files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#discussion_r1116467566
##########
pulsar-functions/proto/src/main/proto/Function.proto:
##########
@@ -165,6 +165,7 @@ message SourceSpec {
bool cleanupSubscription = 11;
SubscriptionPosition subscriptionPosition = 12;
uint64 negativeAckRedeliveryDelayMs = 13;
+ bool skipToLatest = 14;
Review Comment:
@nlu90 - was this discussed on the mailing list? This is a protobuf change. We have a general rule that those go through the PIP process. While there could be exceptions, we should always post about these kinds of changes on the mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on pull request #17214: [improve] configure whether function consumer should skip to latest
Posted by "nlu90 (via GitHub)" <gi...@apache.org>.
nlu90 commented on PR #17214:
URL: https://github.com/apache/pulsar/pull/17214#issuecomment-1447080347
> Hi @nlu90, I'm checking in to follow up with the doc updates. Did you have any plans to update the docs?
@momo-jun Let me know where I can update the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org