You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "KKcorps (via GitHub)" <gi...@apache.org> on 2023/04/17 11:13:00 UTC
[GitHub] [pinot] KKcorps opened a new pull request, #10625: WIP: Make table deletion idempotent
KKcorps opened a new pull request, #10625:
URL: https://github.com/apache/pinot/pull/10625
This PR aims to solve the issue describe in #10603
A new flag `async` is added to the table request.
When set, we trigger table deletion via a background job and only return a job id. The table deletion then proceeds in background via a message handler on controller. The controller now waits for EV to converge and only then triggers deletion for segments and configs.
Pending:
* Tracking progress of deletion from job id
* Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on pull request #10625: Make table deletion idempotent
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on PR #10625:
URL: https://github.com/apache/pinot/pull/10625#issuecomment-1519516805
I am already storing progress in the ZK. That's how the deleteStatusAPI works. Without the message thought how are we supposed to return API response immediately? Other jobs are dealing with IS update only and not EV converge which can take extremely long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10625: WIP: Make table deletion idempotent
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10625:
URL: https://github.com/apache/pinot/pull/10625#issuecomment-1511216924
## [Codecov](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10625](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (60a903a) into [master](https://codecov.io/gh/apache/pinot/commit/0cb456a3b03805dbba5b7b34db7c0d877c08c58d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0cb456a) will **decrease** coverage by `45.95%`.
> The diff coverage is `22.97%`.
```diff
@@ Coverage Diff @@
## master #10625 +/- ##
=============================================
- Coverage 70.37% 24.42% -45.95%
+ Complexity 6495 49 -6446
=============================================
Files 2106 2091 -15
Lines 113004 112711 -293
Branches 17026 16989 -37
=============================================
- Hits 79521 27527 -51994
- Misses 27917 82271 +54354
+ Partials 5566 2913 -2653
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `24.42% <22.97%> (-0.12%)` | :arrow_down: |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ommon/messages/TableDeletionControllerMessage.java](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvVGFibGVEZWxldGlvbkNvbnRyb2xsZXJNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/pinot/controller/BaseControllerStarter.java](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9CYXNlQ29udHJvbGxlclN0YXJ0ZXIuamF2YQ==) | `76.64% <ø> (-6.29%)` | :arrow_down: |
| [...er/ControllerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9Db250cm9sbGVyVXNlckRlZmluZWRNZXNzYWdlSGFuZGxlckZhY3RvcnkuamF2YQ==) | `9.75% <4.87%> (-6.92%)` | :arrow_down: |
| [...ntroller/helix/core/PinotHelixResourceManager.java](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL1Bpbm90SGVsaXhSZXNvdXJjZU1hbmFnZXIuamF2YQ==) | `38.20% <31.81%> (-32.75%)` | :arrow_down: |
| [...oller/api/resources/PinotTableRestletResource.java](https://codecov.io/gh/apache/pinot/pull/10625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFibGVSZXN0bGV0UmVzb3VyY2UuamF2YQ==) | `19.46% <50.00%> (-33.74%)` | :arrow_down: |
... and [1576 files with indirect coverage changes](https://codecov.io/gh/apache/pinot/pull/10625/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #10625: Make table deletion idempotent
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #10625:
URL: https://github.com/apache/pinot/pull/10625#issuecomment-1522523496
You may take a look at how rebalance task is handled `PinotTableRestletResource.rebalance()`. Essentially we want to use a background executor to execute the task and keeps updating the ZK about the progress. Using Helix message will use the Helix executor, but we already have a dedicate executor in controller for such long running tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on pull request #10625: Make table deletion idempotent
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on PR #10625:
URL: https://github.com/apache/pinot/pull/10625#issuecomment-1516031340
@jtao15 can you also take a look please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org